3

Symptom

  • ssh to remote-host works.
  • Simple output like ls works.
  • But more output like find / fails.

Solution

In our case we reduced the MTU to 1200 on eth0 (IPv4) and the symptoms disappeared.

Question

AFAIK there is something call Path MTU Discovery

From Wikipedia:

Many network security devices block all ICMP messages for perceived security benefit

If this is the reason, how to debug this?

1
  • Did any answer help you? If so, you should accept the answer so that the question doesn't keep popping up forever, looking for an answer. Alternatively, you could provide and accept your own answer.
    – Ron Maupin
    Aug 13 '17 at 20:16
4

Path MTU Discovery works by sending packets of various sizes to a target with the DF (Don't Fragment" bit turned on. If a router in the path has a smaller MTU, its behavior should be to send back an ICMP message which states the packet needed to have been fragmented, but the DF bit was set to 1. Often this packet even includes the size of the MTU of the link which required fragmentation.

To debug this, you simply have to do Path MTU discovery, manually.

Send a ping to a target, in my example, I'll use Google's DNS server (8.8.8.8). Set your DF bit in your ping to on, to prevent your ping from being fragmented. Set your packet size to some large number, or the standard MTU of 1500. Note that some ping implementations set the size of just the payload, which means you have to account for the 8 byte ICMP header, and 20 byte IP header.

On a Windows PC, here is the command:

C:\Users\eddie>ping -f -l 1472 -i 1 8.8.8.8

Pinging 8.8.8.8 with 1472 bytes of data:
Reply from 10.200.192.1: TTL expired in transit.
Reply from 10.200.192.1: TTL expired in transit.
Reply from 10.200.192.1: TTL expired in transit.
Reply from 10.200.192.1: TTL expired in transit.

Ping statistics for 8.8.8.8:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss)

Notice I received TTL expired from my first router. That is what I want to see, it tells me nothing in the path between me and my first router required fragmentation.

It would have looked like this if so:

C:\Users\eddie>ping -f -l 1473 -i 1 8.8.8.8

Pinging 8.8.8.8 with 1473 bytes of data:
Packet needs to be fragmented but DF set.
Packet needs to be fragmented but DF set.
Packet needs to be fragmented but DF set.
Packet needs to be fragmented but DF set.

Ping statistics for 8.8.8.8:
    Packets: Sent = 4, Received = 0, Lost = 4 (100% loss)

Either way, do this progressively, increasing the TTL by 1 each time. At some point in the path, you'll encounter the error message above (Packet needs to be fragmented by DF set). When that happens, you'll know exactly what router between you and the target had a smaller link size.

If it happened when the TTL is set to 8, then you can run a traceroute to the target, and the 8th hop will be the router that couldn't fragment the packet.

If at any time, instead of getting Packet needs to be fragmented but DF set or TTL expired in transit, then you then know also where in the path a Firewall exists and is blocking the return ICMP packets.

Edit: Adding Helpful information from the comments:

@YLearn adds:

be aware that the return packets could take a different route back and there could be a problem on the return path rather than a firewall/ACL on the outgoing path

@ guettli (Original Poster) adds:

on linux these options are needed: ping -M do -s 1472 8.8.8.8enter link description here

2
  • 1
    I have no windows here. The-f option means Sets the Do Not Fragment bit on the ping packet. ... on linux these options are needed: ping -M do -s 1472 8.8.8.8
    – guettli
    Mar 24 '16 at 17:07
  • Thank you very much for your explanations. I am just wandering why there is no tool which automates this debugging.
    – guettli
    Apr 4 '16 at 10:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.