You need software that does DPI Deep Packet Inspection). Some companies have made standard rules for some protocols, where in a TCP or UDP packet or some sequence it can identify the protocol or application that is running.
This kind of application realizes an analysis of Layer 7 of a packet, matching the packet content with a huge rule databases.
Like @ojs said, you can tried with nDPI from ntop, test it and see the results.
Cisco also has its own database that can be used with the OPNManager, but requires a license.
