A TCP connection is identified by a pair of sockets (local and remote). Connecting to a remote TCP and handshaking gives you a bidirectional connection. It is like a pair of pipes (one in each direction) between the hosts, and anything going in one end of a pipe comes out the other end.
RFC 793, Transmission Control Protocol explains (see highlights, but the RFC goes into much more detail):
To allow for many processes within a single Host to use TCP
communication facilities simultaneously, the TCP provides a set of
addresses or ports within each host. Concatenated with the network
and host addresses from the internet communication layer, this forms a
socket. A pair of sockets uniquely identifies each connection. That
is, a socket may be simultaneously used in multiple connections.
The binding of ports to processes is handled independently by each
Host. However, it proves useful to attach frequently used processes
(e.g., a "logger" or timesharing service) to fixed sockets which are
made known to the public. These services can then be accessed through
the known addresses. Establishing and learning the port addresses of
other processes may involve more dynamic mechanisms.
The reliability and flow control mechanisms described above require
that TCPs initialize and maintain certain status information for each
data stream. The combination of this information, including
sockets, sequence numbers, and window sizes, is called a
connection. Each connection is uniquely specified by a pair of
sockets identifying its two sides.
When two processes wish to communicate, their TCP's must first
establish a connection (initialize the status information on each
side). When their communication is complete, the connection is
terminated or closed to free the resources for other uses.
Since connections must be established between unreliable hosts and
over the unreliable internet communication system, a handshake
mechanism with clock-based sequence numbers is used to avoid erroneous
initialization of connections.