32

In discussions that have spurred from other questions on this site, I've realised that I don't have a solid understanding of when Path MTU Discovery (PMTUD) is performed.

I know what it does -- discover the lowest MTU on a path from Client to Server).
I know how it does it -- send progressively larger packets with their "Don't Fragment" bit set, and see how big of a packet you can get through without getting a "ICMP Need to Fragment" error.

My question is specifically then, when will a host perform PMTUD?

I'm looking for specific cases. Not just something generic like "when a host wants to discover the path MTU". Bonus points if you can provide a packet capture of a host doing it, or provide instructions for generating such a packet capture.

Also, I am specifically referring to IPv4. I know in IPv6 transient routers aren't responsible for fragmentation, and can imagine that PMTUD happens much more commonly. But for now, I'm looking for specific examples of PMTUD in IPv4. (although if the only packet capture you can put together of PMTUD is in IPv6, I would still love to see it)

Improve this question
3
  • Is PMTUD done from the lowest supported MTU to the highest? Or does the device performing PMTUD try the largest MTU first and then step down by a large increment until the packet passes and then step up in smaller increments, then alternate back and forth until a final determination is made?
    – cpt_fink
    Commented Dec 5, 2014 at 7:25
  • @cpt_fink, there are a few strategies. Modern implementations of the ICMP Fragmentation Needed message include in the ICMP payload itself the MTU of the link for which fragmentation was required. That makes it easy, as the starting host knows right away what the path MTU is. Older implementations have to use various strategies to 'search' for the right MTU to use. Those strategies are outlined in RFC1191 in Section 5. They range from automatically defaulting to IP Minimum (576), to using a table of 'common' MTU's to search more efficiently (see RFC1191 Section 7.1).
    – Eddie
    Commented Dec 5, 2014 at 16:33
  • 2
    This is an interesting question. I was doing some digging on PMTUD and found this. Even though it's old, I decided to answer because I had the exact same question and after some hours of researching I could come up with a pretty decent answer (I guess). I will try to update and support my answer with a packet capture tomorrow, if possible.
    – Filipe Gonçalves
    Commented Jul 1, 2015 at 23:39

4 Answers 4

Reset to default
27

The answer is simple: whenever the host pleases. Really. It's that simple.

The explanation below assumes an IPv4-only environment, since IPv6 does away with fragmentation in the routers (forcing the host to always deal with fragmentation and MTU discovery).

There is no strict rule that governs when (or even if) a host does Path MTU Discovery. The reason that PMTUD surfaced is that fragmentation is considered harmful for various reasons. To avoid packet fragmentation, the concept of PMTUD was brought to life as a workaround. Of course, a nice operating system should use PMTUD to minimize fragmentation.

So, naturally, the exact semantics of when PMTUD is used depend on the sender's operating system - in particular, the socket implementation. I can only speak for the specific case of Linux, but other UNIX variants are probably not very different.

In Linux, PMTUD is controlled by the IP_MTU_DISCOVER socket option. You can retrieve its current status with getsockopt(2) by specifying the level IPPROTO_IP and the IP_MTU_DISCOVER option. This option is valid for SOCK_STREAM sockets only (a SOCK_STREAM socket is a two-way, connection-oriented, reliable socket; in practice it's a TCP socket, although other protocols are possible), and when set, Linux will perform PMTUD exactly as defined in RFC 1191.

Note that in practice, PMTUD is a continuous process; packets are sent with the DF bit set - including the 3-way handshake packets - you can think of it as a connection property (although an implementation may be willing to accept a certain degree of fragmentation at some point and stop sending packets with the DF bit set). Thus, PMTUD is just a consequence of the fact that everything on that connection is being sent with DF.

What if you don't set IP_MTU_DISCOVER?

There's a default value. By default, IP_MTU_DISCOVER is enabled on SOCK_STREAM sockets. This can be read or changed by reading /proc/sys/net/ipv4/ip_no_pmtu_disc. A zero value means that IP_MTU_DISCOVER is enabled by default in new sockets; a non-zero means the opposite.

What about connectionless sockets?

This is tricky because connectionless, unreliable sockets do not retransmit lost segments. It becomes the user's responsibility to packetize the data in MTU-sized chunks. Also, the user is expected to make the necessary retransmits in case of a Message too big error. So, essentially user code must reimplement PMTUD. Nevertheless, if you're up for the challenge, you can force the DF bit by passing the IP_PMTUDISC_DO flag to setsockopt(2).

The bottomline

  • The host decides when (and if) to use PMTUD
  • When it uses PMTUD, it's like a connection attribute, it happens continuously (but at any point the implementation is free to stop doing so)
  • Different operating systems use different approaches, but usually, reliable, connection-oriented sockets perform PMTUD by default, whereas unreliable, connectionless sockets don't
Improve this answer
13

Typically, path maximum transmission unit discovery (PMTUD) happens whenever a host thinks a packet was dropped due to being too large.

This may be in response to ICMP fragmentation required (type 3, code 4) response explicitly indicating the packet was dropped. In typical practice all IPv4 packets are set with the "don't fragment" (DF) flag set, so any packet in excess of the MTU will elicit such a response. IPv6 doesn't support fragmentation at all.

Some routers or host firewalls drop all ICMP often because a naive administrator believes ICMP to be a security risk. Or, some link aggregation schemes may break ICMP delivery. An alternate mechanism to discover the MTU has been exceeded which does not rely on ICMP is proposed in RFC4821.

tracepath is my favorite Linux tool for probing MTU. Here's an example from a host with a 9001 MTU on the LAN, but which must traverse an IPsec VPN to reach 10.33.32.157:

$ tracepath -n 10.33.32.157
 1?: [LOCALHOST]                                         pmtu 9001
 1:  10.1.22.1                                             0.122ms pmtu 1500
 1:  169.254.3.1                                           1.343ms pmtu 1422
 1:  10.255.254.61                                        23.790ms 
 2:  no reply
^C [this host won't return an ICMP port unreachable, so tracepath won't terminate]

The ICMP errors can be observed with tcpdump:

$ sudo tcpdump -p -ni eth0 'icmp and icmp[0] == 3 and icmp[1] == 4'
14:46:57.313690 IP 10.1.22.1 > 10.1.22.194: ICMP 10.33.32.157 unreachable - need to frag (mtu 1500), length 36
14:46:57.315080 IP 169.254.3.1 > 10.1.22.194: ICMP 10.33.32.157 unreachable - need to frag (mtu 1422), length 556

MTU discoveries are cached. In Linux this can be observed and flushed with ip (beware of changes since Linux 3.6):

$ ip route get 10.33.32.157
10.33.32.157 via 10.1.22.1 dev eth0  src 10.1.22.194 
    cache  expires 591sec mtu 1422
$ sudo ip route flush cache
$ ip route get 10.33.32.157
10.33.32.157 via 10.1.22.1 dev eth0  src 10.1.22.194 
    cache

For TCP, exceeding the MTU may be avoided as part of the connection setup. Included in the SYN sent by each end is a maximum segment size (MSS). The TCP header (20 bytes excluding options) and IP header (20 bytes) mean MSS and MTU are related by a difference of 40 bytes.

Here's an example of a connection setup between these two hosts when transferring a large file with scp:

$ sudo tcpdump -p -ni eth0 'host 10.33.32.157 and tcp[13]&2 == 2'
IP 10.1.22.194.45853 > 10.33.32.157.22: Flags [S], seq 634040018, win 26883, options [mss 8961,sackOK,TS val 10952240 ecr 0,nop,wscale 7], length 0
IP 10.33.32.157.22 > 10.1.22.194.45853: Flags [S.], seq 1371736848, ack 634040019, win 26847, options [mss 1379,sackOK,TS val 10824267 ecr 10952240,nop,wscale 7], length 0

In the first packet, the local host proposes an MSS of 8961. This is the configured 9001 MTU, less 40 bytes. The returned SYN/ACK has an MSS of 1379, implying an MTU of 1419. I happen to know in this network the remote host also sent 8961, but the value has been modified by a router since it knows the path includes an internet path (MTU 1500) an overhead from an IPsec tunnel. This router also modified our sent MSS of 8961 to appear as 1419 at the other host. This is called MSS clamping.

So in a sense, PMTUD is happening all the time. In practice, it may actually happen never, if MSS clamping is in place and all traffic is occurring over TCP, or if none of the routers have an MTU smaller than what's configured on the endpoints. Even without MSS clamping it may happen only rarely, when the cache expires.

Improve this answer
1

I found a really good blog about it and PMTUD process is explained in a simple term through packet transfer and TCPDUMP messages, which I found informative.

https://theojulienne.io/2020/08/21/mtu-mss-why-connections-stall.html

Improve this answer
-3

PMTUD is used to calculate the best MSS for TCP sessions. One example is BGP implementation on Cisco or Juniper routers.

http://www.juniper.net/techpubs/en_US/junos12.1/topics/usage-guidelines/routing-configuring-mtu-discovery-for-bgp-sessions.html

Thanks.

Improve this answer
1
  • 2
    I believe he meant "when is it triggered?".
    – Jordan Head
    Commented Dec 4, 2014 at 20:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.