Traceroute seems to succeed but also gives host is unreachable message [duplicate]

Question

I have done a traceroute from the Princeton traceroute server ("https://www.net.princeton.edu/traceroute.html") to the ip address 129.97.208.23. This is the following result I received:

tracing path from www.net.princeton.edu to 129.97.208.23 ...

  traceroute to 129.97.208.23 (129.97.208.23), 30 hops max, 40 byte packets
  1  core-87-router (128.112.128.2)  1.677 ms  0.789 ms  0.593 ms
  2  rtr-border-hpcrc-router.princeton.edu (128.112.12.110)  1.667 ms  1.340 ms  1.383 ms
  3  local1.princeton.magpi.net (216.27.98.113)  4.431 ms  2.903 ms  2.399 ms
  4  216.27.100.18 (216.27.100.18)  2.780 ms  2.578 ms  2.572 ms
  5  et-1-1-0.4079.rtsw.wash.net.internet2.edu (162.252.70.119)  6.580 ms  6.417 ms  6.027 ms
  6  ae-0.4079.rtsw2.ashb.net.internet2.edu (162.252.70.137)  16.091 ms  7.212 ms  7.095 ms
  7  ae-2.4079.rtsw.ashb.net.internet2.edu (162.252.70.74)  6.962 ms  11.326 ms  7.414 ms
  8  ae-20.4079.rtsw.clev.net.internet2.edu (162.252.70.129)  14.775 ms  14.605 ms  14.351 ms
  9  ae-3.4079.rtsw3.eqch.net.internet2.edu (162.252.70.131)  24.254 ms  23.392 ms  23.247 ms
  10  ae-5.4079.rtsw.eqch.net.internet2.edu (162.252.70.162)  46.435 ms  24.698 ms  23.604 ms
  11  ae-0.4079.rtsw.star.net.internet2.edu (162.252.70.111)  23.581 ms  23.540 ms  23.150 ms
  12  205.189.32.241 (205.189.32.241)  33.258 ms  33.525 ms  38.638 ms
  13  205.189.32.40 (205.189.32.40)  33.973 ms  33.849 ms  34.009 ms
  14  66.97.21.2 (66.97.21.2)  50.022 ms  49.974 ms 66.97.21.18 (66.97.21.18)  52.429 ms
  15  66.97.16.229 (66.97.16.229)  49.933 ms 66.97.16.233 (66.97.16.233)  52.488 ms  52.479 ms
  16  66.97.28.66 (66.97.28.66)  27.689 ms  27.733 ms  30.019 ms
  17  * * *
  18  * * *
  19  * * *
  20  * * *
  21  wms.uwaterloo.ca (129.97.208.23)  31.181 ms  33.579 ms  33.218 ms
  22  wms.uwaterloo.ca (129.97.208.23)  30.920 ms !N  30.658 ms !N  33.303 ms !N

According to hop 21 the traceroute is successful. However in hop 21, the same host is repeated and !N is there between the different RTT columns. What does this mean? Is the traceroute successful or not?

Answer 1

This is successful. Your traceroute got to the destination 129.97.208.23 and you got an error message from that destination. Normally the error message for traceroute is ICMP destination port unreachable for the high UDP port that traceroute used. Another common case for public websites is that the final hop does not respond to the the icmp at all (presumably the final hop is a load balancer configured only to repsond to the configured web ports).

The way Unix and MacOS traceroute works is it sends UDP packets with a high destination port (the port number gradually increases as the traceroute runs). The destination server is probably not listening on that port.

Here's the start of a tcpdump taken during a traceroute to that destination:

[iMac:Build/Products/Debug] droot% sudo tcpdump -n icmp or host 129.97.208.23
tcpdump: data link type PKTAP
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on pktap, link-type PKTAP (Apple DLT_PKTAP), capture size 262144 bytes
14:18:09.937737 IP 192.168.0.10.50535 > 129.97.208.23.33435: UDP, length 24
14:18:09.938467 IP 192.168.0.1 > 192.168.0.10: ICMP time exceeded in-transit, length 36
14:18:09.939364 IP 192.168.0.10.50535 > 129.97.208.23.33436: UDP, length 24
14:18:09.939800 IP 192.168.0.1 > 192.168.0.10: ICMP time exceeded in-transit, length 36
14:18:09.939862 IP 192.168.0.10.50535 > 129.97.208.23.33437: UDP, length 24

Note how the destination UDP port increases each request? 33435...33436...33437.

Here's the last few packets:

14:19:07.697298 IP 192.168.0.10.50535 > 129.97.208.23.33497: UDP, length 24
14:19:12.702258 IP 192.168.0.10.50535 > 129.97.208.23.33498: UDP, length 24
14:19:12.823416 IP 129.97.208.23 > 192.168.0.10: ICMP time exceeded in-transit, length 36
14:19:12.824043 IP 192.168.0.10.50535 > 129.97.208.23.33499: UDP, length 24
14:19:12.915354 IP 129.97.208.23 > 192.168.0.10: ICMP time exceeded in-transit, length 36
14:19:12.915433 IP 192.168.0.10.50535 > 129.97.208.23.33500: UDP, length 24
14:19:12.997978 IP 129.97.208.23 > 192.168.0.10: ICMP time exceeded in-transit, length 36
14:19:12.998060 IP 192.168.0.10.50535 > 129.97.208.23.33501: UDP, length 24
14:19:13.080569 IP 129.97.208.23 > 192.168.0.10: ICMP net 129.97.208.23 unreachable, length 36
14:19:13.081186 IP 192.168.0.10.50535 > 129.97.208.23.33502: UDP, length 24
14:19:13.163592 IP 129.97.208.23 > 192.168.0.10: ICMP net 129.97.208.23 unreachable, length 36
14:19:13.163674 IP 192.168.0.10.50535 > 129.97.208.23.33503: UDP, length 24
14:19:13.246592 IP 129.97.208.23 > 192.168.0.10: ICMP net 129.97.208.23 unreachable, length 36

The first one did not get a response. I'm guessing Princeton is using RFC1918 address space for it's router-to-router links, so once you get close your traceroute gives you the * * * output for 4 hops.

Then you get a time-to-live exceeded from the destination. It rejects the packet as it receives it with a TTL of 0.

Then at the very end, when the destination receives a packet with a valid TTL of 1, instead of getting a "ICMP time exceeded" you get a "ICMP net unreachable", which is a bit unexpected (I expected destination port unreachable) but its a traceroute server. Might be some unique filtering.

Bottom line: your traceroute got to the target, trigging an error response from the final IP.