You need software that does DPI Deep(Deep Packet Inspection). Some companies have made standard rules for some protocols, where in a TCP or UDP packet or some sequence, it can identify the protocol or application that is running.
This kind of application realizes an analysis of Layer 7 of a packet, matching the packet content with a huge rule databasesrules database.
Like @ojs said, you can triedtry with nDPI from ntop, test it and see the results.
Cisco also has its own database that can be used with the OPNManager, but requires a license.