What's the benefit of IPSec over TLS and DTLS ? If someone's goal is to assure authentication, integrity and confidentiality, can't he/she simply encrypt the content with the two last protocols (on top of TCP and UDP, respectively) ? Why do I only see IPSec used in VPNs?
At a high level, they are equivalent. They are both "Secure Communication" protocols which create a "tunnel" between two end points. Data transferred within this tunnel is protected with Confidentiality, Integrity, and Authentication.
In practice, TLS/SSL/DTLS & IPSec (and SSH!) are all considered equally secure as protocols -- it's more the choice of algorithms used within the protocols that make one more secure than the other. (i.g., 3DES IPsec is less secure than AES-128 TLS, which is less secure than AES-256 IPsec, and so on)
But if you dig into the inner workings of each, there is a critical difference between the two (although, the answer probably belongs in Information Security or Crypto Stack Exchange).
Traditionally this is done in two separate operations -- one operation for Encryption, and the other operation for the MAC. The issue then, is if there are two operations... which one should be done first?
TLS and IPsec picked different strategies:
TLS/SSL does MAC-then-Encrypt -- a MAC is calculated over the plaintext, the resulting digest is appended, and then both the plaintext and the digest are encrypted to produce the cipher text.
IPsec does Encrypt-then-MAC -- The plaintext is encrypted first. Then a MAC is calculated over the cipher text and the resulting digest is appended.
The strategy TLS/SSL chose ended up being the less secure of the two. Mainly because on the receiving side, you had to perform the decryption operation first before you can check to see if the message was tampered with. This violates what one white-hat hacker calls the The Cryptographic Doom Principle.
Hence, if you peel back many layers, you come to this conclusion: IPsec is more secure than TLS/SSL.
The next question is ... why isn't it used everywhere instead of TLS?
Because of the how the IPsec and IKEv1 negotiation work, if you have to static end points connecting to each other consistently, you can use Main Mode which doesn't infer any significant security risk.
But if you have two non-static end points, like say... web browsers from all over the world, or remote employees connecting from home / coffee shops / airports / hotels, the IKEv1 negotiation uses Aggressive Mode which unfortunately does leak potentially sensitive information.
(I wrote about main mode and aggressive mode on this stack exchange elsewhere, I'll try to dig up the link and add them here)
TLS doesn't have this problem... as long as you are using the current versions of SSL/TLS, the negotiation whether between remote hosts or static hosts is identical.
Moreover, IPsec typically requires additional software running on end-hosts (VPN Clients, etc). Where as implementations of TLS/SSL ship ubiquitously with all OS's and Web Browsers. This makes the "cost" of running TLS/SSL software much lower than having to install IPsec software.
Which finally brings us to your question:
In scenarios where the end points connecting are static, the conventional approach is to build a site-site VPN using IPsec, since it is more secure.
In scenarios where the end points connecting are not static, the conventional approach is to build a "secure tunnel" using TLS/SSL, since it is more convenient.
This became the go-to standard approach for how to use TLS/SSL or IPsec to enable secure communication.
That said... this conventional approach was derived from the history of these protocols. I want to add a couple points to account for the modern day:
- The current recommendation for TLS/SSL is to use AEAD ciphers which do Encryption/MAC in one step, in the correct order. If you are using an AEAD cipher, there is no security gap between TLS/SSL
- The current recommendation for IPSec is to use IKEv2, which does not leak sensitive information for remote clients
They both provide authentication and confidentiality. The difference is TLS is configured by the application. In other words your application needs to be configured to use it.
An IPSec VPN, on the other hand, is configured at the network layer and creates a tunnel for the application. The application is unaware that a VPN is being used.
You're mixing two different things:
- VPN as a tunnel provider versus SSL/TLS for application-specific encryption
- IPsec over TLS/DTLS for other reasons
The main point for a VPN is to create a tunneled link where you can pass private traffic in a secure way and with private addressing. A VPN can be used with any type of traffic, without specific setup on the application layer. IPsec provides that but may suffer from accidental or deliberate filtering - inside a TLS/DTLS wrapper, it might be less conspicuous or 'exotic' and pass more easily. So, some remote-access products offer a TLS/DTLS fallback when plain IPsec fails.
Also, a VPN-as-a-service provider could terminate TLS/DTLS user tunnels for multiple customers on the same backend/central infrastructure, yet use different inner IPsec tunnels to terminate a second time on customer gateways. That way, the customers' networks can be kept completely separated from that of the service provider and from each other (dual DMZ).