I would like to understand the difference between overlay networks and VPN? As far as I understand, they are based on virtual connection establishment between endpoints. How do they differ then?
An overlay network may include encryption of payload. VPN's are a type of overlay network that usually includes payload encryption.
The VPN security model provides: [Wikipedia]
Confidentiality such that even if the network traffic is sniffed at the packet level (see network sniffer and Deep packet inspection), an attacker would only see encrypted data.
Some overlay networks include the term VPN but do not meet the above requirement of the VPN security model. An example would be 'MPLS VPN'. That is why you see many MPLS VPN customers apply encryption using their edge gateways.
The term "VPN," or Virtual Private Network, has become almost as recklessly used in the networking industry as has "QoS" (Quality of Service) to describe a broad set of problems and "solutions," when the objectives themselves have not been properly articulated. This confusion has resulted in a situation where the popular trade press, industry pundits, and vendors and consumers of networking technologies alike generally use the term VPN as an offhand reference for a set of different technologies.
So what is a virtual private network? As we have discussed, a VPN can take several forms. A VPN can be between two end systems, or it can be between two or more networks. A VPN can be built using tunnels or encryption (at essentially any layer of the protocol stack), or both, or alternatively constructed using MPLS or one of the “virtual router” methods. A VPN can consist of networks connected to a service provider’s network by leased lines, Frame Relay, or ATM, or a VPN can consist of dialup subscribers connecting to centralized services or other dialup subscribers.