In RFC6146 stated:

Any protocols that protect IP header are incompatible with NAT64. This implies that end-to-end IPsec verification will fail when the Authentication Header (AH) is used (both transport and tunnel mode) and when ESP is used in transport mode.

Why does transport mode ESP can't be used in NAT64 even though both ESP transport and tunnel mode doesn't sign the new IP header?

  • Did any answer help you? If so, you should accept the answer so that the question doesn't keep popping up forever, looking for an answer. Alternatively, you can post and accept your own answer.
    – Ron Maupin
    Dec 16, 2020 at 23:36

2 Answers 2


Transport mode cryptographically verifies the IP addresses. NAT must manipulate the IP addresses to provide connectivity. That's why they're not compatible.

Tunnel mode works through NAT. Transport mode doesn't.

  • Wait I thought the one that matter is the protocol used, not the transport/tunnel mode, like in AH the IP header is signed, thus preventing NAT from manipulating it. Transport Mode ESP on the other hand, "move" the IP header to the front of ESP header but not signing it. Isn't the NAT is supposedly able to manipulate the IP header? Nov 4, 2020 at 1:34
  • 3
    It's more a function of NAT64 not being NAT. It's more like a proxy than NAT. It's translating one protocol to another, not simply changing addresses in a header. Keep your IPSec all IPv4 or all IPv6, trying to mix the two will only make a mess.
    – Ricky
    Nov 4, 2020 at 1:53
  • ESP in transport mode does NOT cryptographically protect the IP addresses. Only AH does this (no matter which mode).
    – ecdsa
    Nov 4, 2020 at 8:51

When you use a connection involving NAT64, your topology looks like this:

+-----+   +-------+   +-----+
| IP6 |===| NAT64 |---| IP4 |
+-----+   +-------+   +-----+

One peer is using IPv6 addresses, one peer is using IPv4 addresses and the NAT64 gateway translates between these addresses.

Now let's look at a TCP connection. RFC793 says in 3.1. Header Format the following about the checksum field in the TCP header:

The checksum also covers a 96 bit pseudo header conceptually
prefixed to the TCP header.  This pseudo header contains the Source
Address, the Destination Address, the Protocol, and TCP length.
This gives the TCP protection against misrouted segments.  This
information is carried in the Internet Protocol and is transferred
across the TCP/Network interface in the arguments or results of
calls by the TCP on the IP.

With a normal TCP stream the NAT64 gateway is able to recalculate the checksum and send out valid TCP datagrams in either direction.

But in an ESP datagram the TCP header is encrypted, so it can't be recalculated. Therefore the decrypted TCP datagrams become invalid after decryption because there are different addresses at each peer.

The same is true for UDP. Although for IPv4 the UDP checksum is not mandatory, it must be set to 0 if not used, and it is required to be used for IPv6, so there will be a checksum on the UDP datagrams.

Edit: changed text about UDP according to Ron Maupins comment.

  • "at least for IPv4 the UDP checksum is not mandatory." That is true, but it must be set to 0 if not used, and it is required to be used for IPv6, so there will be a checksum on the UDP datagrams.
    – Ron Maupin
    Nov 4, 2020 at 16:00
  • So, what's the difference between ESP in tunnel and transport mode? Both transport & tunnel mode ESP doesn't add new TCP header in front of ESP header. Nov 5, 2020 at 11:10
  • In tunnel mode you have two IP headers, the outer one that gets NATed and the inner one that is encrypted and remains unchanged. In transport mode you have only the outer IP header. The TCP header belongs to the inner IP-Header with tunnel mode and to the outer IP-Header with transport mode. Nov 5, 2020 at 16:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.