What's the benefit of IPSec over TLS and DTLS ? If someone's goal is to assure authentication, integrity and confidentiality, can't he/she simply encrypt the content with the two last protocols (on top of TCP and UDP, respectively) ? Why do I only see IPSec used in VPNs?

  • Unfortunately, questions about protocols above OSI layer-4 are off-topic here. You could try to ask this question on Server Fault for a business network.
    – Ron Maupin
    Oct 2 '21 at 13:50
  • This question was posted 2 months ago. There have been zero close votes from the community. The question comes up again today.... and gets Closed by @RonMaupin. Care to talk me through that decision process, Ron? Are we considering IPsec "above L4" now? Will you be closing all the other IPsec questions as well?
    – Eddie
    Oct 2 '21 at 19:21
  • Both protocols in the question are above OSI layer-4; they are carried by layer-4 transport protocols. I do not necessarily see every question, and I closed this question when I saw it because it is off-topic here. @Eddie, you should know that this is not the place to argue such things; we have Network Engineering Meta for that.
    – Ron Maupin
    Oct 2 '21 at 19:27
  • @RonMaupin in what universe is something, transmitted as IPv6 extension header, above layer 4 or carried by layer 4 transport protocol????
    – Effie
    Dec 27 '21 at 22:04
  • @Effie, TLS and DTLS are not IPv6 extension headers.
    – Ron Maupin
    Dec 27 '21 at 22:06

At a high level, they are equivalent. They are both "Secure Communication" protocols which create a "tunnel" between two end points. Data transferred within this tunnel is protected with Confidentiality, Integrity, and Authentication.

In practice, TLS/SSL/DTLS & IPSec (and SSH!) are all considered equally secure as protocols -- it's more the choice of algorithms used within the protocols that make one more secure than the other. (i.g., 3DES IPsec is less secure than AES-128 TLS, which is less secure than AES-256 IPsec, and so on)

But if you dig into the inner workings of each, there is a critical difference between the two (although, the answer probably belongs in Information Security or Crypto Stack Exchange).

Confidentiality is provided with Symmetric Encryption. Integrity is provided with a MAC, or Message Authentication Code.

Traditionally this is done in two separate operations -- one operation for Encryption, and the other operation for the MAC. The issue then, is if there are two operations... which one should be done first?

TLS and IPsec picked different strategies:

  • TLS/SSL does MAC-then-Encrypt -- a MAC is calculated over the plaintext, the resulting digest is appended, and then both the plaintext and the digest are encrypted to produce the cipher text.

  • IPsec does Encrypt-then-MAC -- The plaintext is encrypted first. Then a MAC is calculated over the cipher text and the resulting digest is appended.

The strategy TLS/SSL chose ended up being the less secure of the two. Mainly because on the receiving side, you had to perform the decryption operation first before you can check to see if the message was tampered with. This violates what one white-hat hacker calls the The Cryptographic Doom Principle.

Hence, if you peel back many layers, you come to this conclusion: IPsec is more secure than TLS/SSL.

The next question is ... why isn't it used everywhere instead of TLS?

Because of the how the IPsec and IKEv1 negotiation work, if you have to static end points connecting to each other consistently, you can use Main Mode which doesn't infer any significant security risk.

But if you have two non-static end points, like say... web browsers from all over the world, or remote employees connecting from home / coffee shops / airports / hotels, the IKEv1 negotiation uses Aggressive Mode which unfortunately does leak potentially sensitive information.

(I wrote about main mode and aggressive mode on this stack exchange elsewhere, I'll try to dig up the link and add them here)

TLS doesn't have this problem... as long as you are using the current versions of SSL/TLS, the negotiation whether between remote hosts or static hosts is identical.

Moreover, IPsec typically requires additional software running on end-hosts (VPN Clients, etc). Where as implementations of TLS/SSL ship ubiquitously with all OS's and Web Browsers. This makes the "cost" of running TLS/SSL software much lower than having to install IPsec software.

Which finally brings us to your question:

  • In scenarios where the end points connecting are static, the conventional approach is to build a site-site VPN using IPsec, since it is more secure.

  • In scenarios where the end points connecting are not static, the conventional approach is to build a "secure tunnel" using TLS/SSL, since it is more convenient.

This became the go-to standard approach for how to use TLS/SSL or IPsec to enable secure communication.

That said... this conventional approach was derived from the history of these protocols. I want to add a couple points to account for the modern day:

  • The current recommendation for TLS/SSL is to use AEAD ciphers which do Encryption/MAC in one step, in the correct order. If you are using an AEAD cipher, there is no security gap between TLS/SSL
  • The current recommendation for IPSec is to use IKEv2, which does not leak sensitive information for remote clients

They both provide authentication and confidentiality. The difference is TLS is configured by the application. In other words your application needs to be configured to use it.

An IPSec VPN, on the other hand, is configured at the network layer and creates a tunnel for the application. The application is unaware that a VPN is being used.

  • Is it accurate to say that the only benefit of IPSec over TLS is that it requires a single setup instead of a setup per application? Jul 31 '21 at 21:58
  • If by "TLS" you mean "SSL" (https, etc.), then yes, the application has to handle it per connection. However, there are numerous VPN systems that can use TLS/DTLS (i.e. "SSL VPN" software) in the same manner you ascribe to IPSec.
    – Ricky
    Aug 1 '21 at 5:46

You're mixing two different things:

  • VPN as a tunnel provider versus SSL/TLS for application-specific encryption
  • IPsec over TLS/DTLS for other reasons

The main point for a VPN is to create a tunneled link where you can pass private traffic in a secure way and with private addressing. A VPN can be used with any type of traffic, without specific setup on the application layer. IPsec provides that but may suffer from accidental or deliberate filtering - inside a TLS/DTLS wrapper, it might be less conspicuous or 'exotic' and pass more easily. So, some remote-access products offer a TLS/DTLS fallback when plain IPsec fails.

Also, a VPN-as-a-service provider could terminate TLS/DTLS user tunnels for multiple customers on the same backend/central infrastructure, yet use different inner IPsec tunnels to terminate a second time on customer gateways. That way, the customers' networks can be kept completely separated from that of the service provider and from each other (dual DMZ).

  • "a service provider could terminate multiple clients on the same backend, yet use different inner tunnels to terminate a second time on client gateways" I don't know what service provider, backend and client gateways mean in this context. Could you give an example? Jul 31 '21 at 22:02
  • Default ports can be filtered for any application. IPSec (ISAKMP/500) is just a very common target. TLS has the advantage of looking just like any other SSL ("HTTPS") connection, and often runs on port 443, making it even harder to identify. DTLS is UDP, so back to very ease to spot. (and at least for Cisco, the default port is 10000)
    – Ricky
    Aug 1 '21 at 5:51
  • @AlanEvangelista I was alluding to a dual-DMZ construct. The service provider provides/sells the user-side service to their clients. The backend is the central infrastructure where the outer D/TLS user tunnels terminate at. The inner IPsec tunnels then terminate at various client gateways which are located close to or inside the client networks,
    – Zac67
    Aug 1 '21 at 7:21
  • @AlanEvangelista I've rephrased the section somewhat - it's an approach we're actually considering for clean client/customer separation.
    – Zac67
    Aug 1 '21 at 7:33

DTLS cannot hide topology while IPSEC can.

Not the answer you're looking for? Browse other questions tagged or ask your own question.