I'm researching various tunneling technologies and I have a fairly good understanding covering some of the basics. In a site-to-site VPN the edge routers could appear as only one hop away. I've also read that it's possible to make it appear as a flat network i.e. everything on the same LAN. This is the piece i am missing, can someone help me to understand how this is possible?
Some tunneling technologies provide Ethernet over IP services. For instance, research about these topics:
Yes it is possible, and Everton gave you some features that help you do that, but...
...please don't do it !
I know a single LAN over two (or more) distant sites seems nice for machine migration (it can keep the same IP) or auto-discovering protocols, but by bridging two remote LAN you create a single failure domain.
By bridging remote LAN, you will forward every broadcast, every multicast, every broadcast storm, every shitty multicast packet that will triggeer some bug on oh-so-bugged multicast stacks on some hosts... Even for legitimate broadcast it is bad, because all of these have to be treated by everyone before being considered irrelevant by 99% of your hosts. That costs CPU. LAN do not scale !! (Hint : the "L" stands for "Local" :) )
See Ivan Pepelnjak's blog post for more detailed explanation.
If you need to move a host without having to change it's IP address, consider using a mechanism where the host (or its hypervisor if applicable) announces a /32 route to your router. Calico does it with BGP, for instance.
Theres a few ways to do it.
One easy way is just to build a virtual ethernet link. The VPN software creates a virtual ethernet interface on both ends and forwards frames between them. You can then use bridging to connect the virtual ethernet interfaces to real ones. I have successfully done this using both vtun and openvpn. I preffer openvpn because it has built-in bridge functionality that allowsone openvpn instance to serve multiple clients. There are also similar solotions from the proffesional networking world but I don't have personal experiance with them.
The upside of this approach is that pretty much everything (unless it's very latency sensitive) that works on a LAN just works. Broadcasts get to where they need to go, hosts can seamlessly move from one site to another without changing their addresses.
The downside of this approach is that the overhead can be relatively high. Firstly the growth in packet size from the extra headers means that full size packets from your LAN won't fit in a normal internet MTU. So either they have to be fragmented somehow or the network you run your tunnel over needs to support jumbo frames. Secondly a load of pointless broadcast chatter can end up going down the tunnel. Thirdly it can be difficult to keep local traffic local and avoid triangle routing.
There are fancier systems too that emulate some aspects of an ethernet network while trying to avoid some of the above problems but again I have no personal experiance with them.
The main thing that makes things "appear on the same LAN" are broadcast messages of one type or another (depending on the type of LAN and what exactly we mean by "appear on"). Specifically, in the case of Ethernet (as an example), all nodes that can receive and respond to data-link layer broadcast messages (e.g., ARP) will appear to be on the local LAN (more properly, the same broadcast domain).
In order to create the illusion of a node being on the local LAN, all a device or service has to do is handle two-way transmission of link-layer broadcast messages and responses. So two broadcast domains that are joined by a VPN connection using devices that accept and encapsulate and de-encapsulate all broadcasts and responses will effectively become one broadcast domain.
Of course the network layer addressing will have to arranged accordingly.