You need software that does DPI Deep Packet Inspection). Some companies have made standard rules for some protocols, where in a TCP or UDP packet or some sequence it can identify the protocol or application that is running.

This kind of application realizes an analysis of Layer 7 of a packet, matching the packet content with a huge rule databases.

Like @ojs said, you can tried with [nDPI][1] from [ntop][2], test it and see the results. 

Cisco also has its own database that can be used with the OPNManager, but requires a license.


  [1]: https://github.com/ntop/nDPI
  [2]: http://www.ntop.org/products/deep-packet-inspection/ndpi/