I am familiar with ARP and routing, but I think I am failing to put two and two together here. On an IPv4 packet we have the source IP address, which allows any router to send the response back to the sender. However, when the response is received, how does the router know which machine sent the original packet request to send it back? We have an ARP table for IP -> MAC address, but the source address on the IPv4 packet is the public IP address not an internal one.
Clarifying your question here, you have a network device which is performing NAT (Network Address Translation) or possibly PAT (Port Address Translation). When the reply packet comes in, how does the NAT/PAT device know how to transform and forward the packet?
The answer is that the NAT/PAT device maintains STATE. It has a "NAT Translation Table". Here's an example of the "show ip nat translations" table on a Cisco ASA:
Router#show ip nat translations Pro Inside global Inside local Outside local Outside global udp 22.214.171.124:1220 192.168.1.95:1220 126.96.36.199:53 188.8.131.52:53 tcp 184.108.40.206:11012 192.168.1.89:11012 220.127.116.11:23 18.104.22.168:23 tcp 22.214.171.124:1067 192.168.1.95:1067 126.96.36.199:23 188.8.131.52:23
A TCP or UDP socket consists of 5 pieces of information:
protocol (udp or tcp) source ip source port destination ip destination port
Lets take the first translation in the table above. The socket appears to be initiated outbound through a port-address translation (PAT).
protocol = udp (always unchanged) source ip = inside local ip = 192.168.1.95 source port = inside local port = 1220 destination ip = outside local ip = outside global ip (unchanged) = 184.108.40.206 destination port = outside local port = outside global port (unchanged) = 53
The only thing that needs to change is our source ip (we can't route RFC1918 address space on the internet) and possibly our source port.
We translate inside local 192.168.1.95 to inside global 220.127.116.11 (from our NAT pool). Then we check if port 1220 is available for NAT pool ip 18.104.22.168. It is! So our inside local port 1220 can also be unchanged for our inside global port 1220.
And that brings us to wholly defining our translation, which we store in our translation table:
Pro Inside global Inside local Outside local Outside global udp 22.214.171.124:1220 192.168.1.95:1220 126.96.36.199:53 188.8.131.52:53
Then we get a reply packet:
protocol udp source ip 184.108.40.206 source port 53 destination ip 220.127.116.11 destination port 1220
We look that up in our translation table and find a match. So we translate each "global" to the "local". In our case the only change is from "inside global 18.104.22.168" to "inside local 192.168.1.95" resulting in:
protocol udp source ip 22.214.171.124 source port 53 destination ip 192.168.1.95 destination port 1220
Then we happily forward the translated packet to our destination: 192.168.1.95 !
Two other general observations:
NAT is very similar to stateful firewalling. So similar that most vendor products that do one generally do the other.
NAT devices have a finite state table size, so these entries need to timeout. A very busy NAT (or stateful firewall) device can have its table max out, resulting in state table entries being cycled quickly and udp/tcp sockets being interrupted.
A router is not taking care of responses. When a "client" host sends a request to a "server" host, it's the responsibility of the "server" to send a response to the source of the request. A router in the network layer has no concept for request and response, or "client" and "server" - those only exist in the application layer.
A router's task is to forward a packet toward its destination address. The source address is only used when the router encounters a problem with the packet, e.g. when the TTL is expired. Then it sends an ICMP time exceeded message to the source and drops the packet.
To forward a packet a router queries its routing table for the destination address. The according routing entry tells the router which interface and gateway to use.
On MAC-based interfaces running IPv4, the router ARPs the gateway and uses the resulting MAC address for the frame it encapsulates the packet in.
On the final routing hop, the destination shares the MAC segment with the router, so the router uses the destination IP as gateway - delivering the packet to the destination itself.
The very frequent translation between private (LAN) and public (Internet) addresses - NA(P)T - breaks this concept. A NAT router needs to be stateful and remember the connections it is modifying. When a private IP client connects to a public IP server, the NAT router substitutes its own public IP address for the LAN client's source address. At the same time, it stores the source/destination IPs and transport-layer source/destination ports in its NAT table.
When a corresponding packet arrives from the former destination (the server), the destination IP and port are translated backwards, so that the packet can be forwarded to the client.
So, in order for a public IP packet to reach a private destination, the connection has to originate from that private destination. This kind of NAT is usually called source NAT.
There's also a NAT variant for allowing inbound public connections to a server using a private IP address. For this, the NAT router "forwards" one of its transport-layer ports to the LAN address - it translates any packet addressed that way to the private IP. This is called destination NAT, reverse NAT, port forwarding, or sometimes virtual IP.