When I write of "usual" here, I am thinking of your average consumer WLAN-NAT router with a sane configuration, or some simple Linux networking with default settings. As usual, this can be made as complex or complicated as necessary. As the question is very basic, this seems to make most sense to me instead of going straight for any more complicated enterprise-level NAT solutions.
You have already accepted an answer, but let me try to directly address the question you asked:
How does NAT decide which connections are inbound, and which are outbound?
The basis for the decision (for any decision in a router) is a set of rules in some form or fashion. In this case, for each of the interfaces involved (i.e., the internal LAN interface vs. the external WAN/uplink interface), the administrator will have implemented rules. Those rules are quite different, i.e. the rules for a LAN interface look wildly different from those of a WAN interface.
Knowing where a packet comes from and where it goes to is the bread and butter of what a router does.
Let me start with an
Example
Wikipedia's NAT page has a lot of text on this issue, but a simple case (a simple company LAN vs. a single DSL uplink) this is what happens:
- The client PC tries to initiate a HTTP connection to an internet-based server, for example 198.51.100.20. The PC itself has a non-routed address like 192.0.2.2. The cheap DSL router has two interfaces, one internal (192.0.2.1) and an external (203.0.113.10, very likely changing often and provided through some local link protocol by the DLS provider). So the PC sends a SYN packet to 198.51.100.20:80 via its default gateway, which is 192.0.2.1.
- The router picks up the packet at its interface 192.0.2.1, like it also would do if there were no NAT involved at all. It has been configured to do NAT on this interface, so it proceeds to do these things:
- Invent a new port number, which is unused so far. For example 12345.
- Change the "sender" address in the IP header to 203.0.113.10.
- Remember the original sender port number in the TCP header (provided by the PC), let's call it 4321.
- Change the TCP header to contain the 12345 sender port number.
- Add an entry (12345; 192.0.2.2; 4321) in its NAT translation table.
- Send the packet along on its merry way to its own uplink/gateway.
- 198.51.100.20 eventually receives the packet, notices that it is a SYN ("establish new connection") and sends a SYN-ACK message back to the sender. From its point of view, this is the IP address 203.0.113.10, with the TCP destination port of 12345.
- The router receives this packet on its WAN interface. The WAN interface has been configured to resolve NATted addresses like this. The router then...
- ...checks its NAT translation table, finds the entry...
- ...modifies the packet to a destination of 192.0.2.2...
- ...fixes the TCP destination port back to 4321...
- ...and sends it alongs its merry way (on the LAN interface).
- The PC receives the packet and does not see anything about the NAT procedure. The packet looks just like if 198.51.100.20 had sent it, as if the NAT router was not there at all.
At no point at all does the topic of a "connection" appear. The NAT router (in its simplest form) does not need to care about the content of the messages. It cares about the IP addresses and the ports of the sender and receiver, but nothing else. (Granted: this is likely skipping over all kind of security- and performance-related issues; but this is about the very basic principle like the one at hand in this question.)
So how does the router know?
The router does not need to know about "connections" at all. In fact, similar procedures like described for TCP exist for the connectionless UDP protocol (UDP hole punching), or could, really, be implemented for any protocol that has something like port numbers in the transport layer.
The reason why the router needs to know the transport-level protocol (TCP, UDP, ...) in regards to NAT is mainly that the ports themselves are not part of IP; and ports are what makes the "hack" that (this kind of) NAT is, easily possible.
So, to your question:
How does NAT decide which connections are inbound, and which are outbound?
Outbound connections are per definitions those that start with a SYN packet (or an initial UDP punch in the case of UDP) appearing at the LAN interface. Calling them "connection" in the case of NAT is a bit much; they end up simply as a temporary entry in a NAT translation table (plus whatever security/performance additions the individual NAT router might employ as well).
Inbound connections do not exist in the scenario I used in the answer so far.
There are of course variants of NAT that do this; for example you can statically identify a port on the WAN interface of the router with a specific IP:PORT on the LAN interface, which makes it possible to run a server inside your NATted LAN. This is also often supported by cheap consumer DSL/WLAN routers. And with "real" routers, you are obviously able to configure them in whichever form or fashion you like.
Further inbound/outbound IP packets are not different from the ones given in the example. Once the initial SYN handshake has been done and the router has the entry in its translation table, it will pass through (with the same translation as explained in the example) all further packets in both directions.
If, in the context of a thus established TCP connection, the server wants to send data to the client (which it is perfectly possible - TCP is bidirectional), these are just further IP packets, as far as the NAT router is concerned. It will not really care that much about the contents of those packets (i.e., whether they contain certain payloads, or are just "management" packets of TCP or whatever).
At no point does the router somehow "close the tube" as you put it. Obviously, the router will have some notion of when it can clear out the entry from the translation table (probably when it notices a FIN handshake which ends the connection, or by some timeout or some error state), but from start to finish it is one continuous affair.