You have hit upon a big problem with NAPT. The NAPT device must maintain a state for connections. It has a table to match incoming packets with outgoing packets.
IP was designed to be a stateless, end-to-end connection. Only the endpoints are supposed to maintain the state. NAPT breaks that paradigm. The connections now depend on an intermediate device to maintain a state. If you have several routers with different paths, and you have asymmetric traffic (return packets come back on a different path), which is not uncommon, then the connection is broken because the return router does not have the state table created by the outbound router.
NAPT is a problem for transport-layer protocols other than TCP, UDP, and ICMP, and there are application-layer protocols that it also breaks. It was created to extend the life of IPv4 until we can all move to IPv6, which has enough addresses to restore the original IP paradigm of every host having a unique address, and the hosts are the ones to maintain the state, not intermediate devices.