Communication between public IPv4 and private IPv4 requires a form of NAT working in both directions. However, you can destination NAT the public IP address of a router to the private IP of a specific LAN host by address alone, ie. on the network layer.
In the reverse direction (private to public source NAT) you'd need to have a 1:1 address relationship as well - the NAT router could provide Internet access for just a single private client. Without looking at the transport layer, the router wouldn't be able to map responses from the Internet back to the correct initiator if there were more than one.
Essentially, source and destination NAT become the same thing: the NAT router substitutes its public address for the (single) private source address for outgoing packets. For incoming packets, it replace its public address by the private address it maps to.
For completeness, there may be scenarios where you don't need any translation between public and private address space, e.g. with a DMZ running in public address space. Since the only difference between private and public addresses is convention, the DMZ router just needed a working route for the private network and vice versa.
Regarding communicate ... by layer 3 alone: Host applications communicate using application-layer protocols encapsulated by transport-layer protocols (encapsulated by network-layer protcols, ...). They cannot communicate "by layer 3 alone". (Depending on your definition of "communicate".)