I've been reading up on Deep Packet Inspection for Software Defined Networks. However, I know that some applications use encryption to evade Deep Packet Inspection. Is there any alternative (such as some Machine Learning algorithm) that would work better with encrypted packets?
For the actual payload inspection you need to break the encryption. That is the only way to detect drive-by malware downloads and similar threats.
The usual way that works is the same way as a man-in-the-middle attack: the server-side encryption is terminated at the inspecting firewall, the firewall re-encrypts the client-side connection and passes data in between.
With the usual certificate-based encryption, the firewall has only access to the server's original, private key in very few scenarios. That means the firewall has to create new, on-the-fly keys and certificates to fit each server connection. Of course, it can't use the original CA key to sign these certificates either, so you need a (sub) CA certificate on the firewall that are distributed to and trusted by the clients. A common method is to distribute a root CA certificate to the clients and use a subordinate signing certificate & private key on the firewall.
As you point out, traditional DPI methods have limited ability to deal with encrypted traffic completely. They can still address encrypted traffic at a surface level at the very least, but it does tend to "cripple" their functionality in many ways.
The new trend in security of this type is Network Traffic Analysis (NTA). Just as many companies are far less concerned today about specific customer data and instead find more value in metadata, NTA tends to care less about the actual data that is being passed and focuses on the behavior of the traffic. Also, similar to DPI, the exact nature of how the traffic is analysed varies from vendor to vendor.
NTA uses behavioral analysis to sort through network traffic and identify dangerous or suspicious traffic based on what the traffic/host does on the network, rather than the exact nature of the data that is passed. This allows NTA to potentially classify malicious traffic (and possibly take action) even when the data is encrypted.
While a newer technology, there are already at least a dozen or two vendors that have their own NTA solution (including several of the big networking companies). It is a relatively new market and there are dramatic differences in the vendor offerings. Some of the techniques used by these vendors include machine learning (supervised and/or unsupervised), signature-based detection, statistical analysis, heuristic analysis, and relatively simple pattern matching among other methods of detection and classification.
Other ways solutions from these vendors differ is how data is collected (span ports/taps, inline, netflow, etc), where the solution exists on the network (centrally, sensors deployed throughout he network, in the cloud or SaaS, etc), whether the solution is hardware or software based, if the solution takes action to remediate threats automatically or requires manual intervention, and of course costs/licensing. While most vendors are selling NTA as a subscription (based on size of organization or aggregate bandwidth), there are at least a couple vendors where you buy a traditional "perpetual" license and pay annual support.
In general DPI engines uses different techniques for detect encrypted traffic, this depends in the nature of the application, for example Skype uses encrypted traffic but is easy to detect them by correlating the first 3 or 4 packets. On the other hand, the most advance technique that I have seen is to verify the dispersion of the bytes on the network flow (frequency analysis), this technique is very efficient but you need to have a lot of packets of the flow to verify the dispersion of the bytes.