I do realize that most LANs sit behind a device running NAT, which translates the users private IP address to a public IP address. My question is still if you are trying to connect to a private device, how does the internet know which private IP you are connecting to, since many LANs have the possibility of using that IP address? Is it that you can only connect using the public IP address?
In order for some device on the internet to connect inbound to a device on your private network you must use a NAT configuration. On residential equipment like a cable modem/router this is often referred to as port-forwarding. Think of it this way. If you are using a single public IP address and sharing it between several private addresses, you must tell the cable modem/router that if I receive a connection on the public IP address on TCP port 80, then that should go to the private address of x.x.x.x on port 80 (or any port you want really). This gives the modem/router enough information to successfully deliver the packets to the internal device from a device out on the internet.
Is it that you can only connect using the public IP?
Basically, yes. Every IP address connected to the public Internet should be unique*. One of the premises of IP is that every device has a unique IP address and end-to-end connectivity. NAT breaks that in order to try to stretch IPv4 until IPv6 is ubiquitous, which will restore unique addressing and end-to-end connectivity.
*There are also reasons to share public addresses. Anycast uses this to provide more localized services by having multiple devices with the same address in different locations. IP routing will direct packets to the nearest (according to the network protocol) device.