Yes, different applications can bind to the same port on different transport protocols. They can also open the same port on the same protocol but different IP addresses.
As the packet works it's way up the processing chain various bits of information are looked at.
Lets say a frame containing an IP packet containing a UDP datagram comes in on an Ethernet card.
First the host will look at the destination MAC address to see if it matches either it's own address or an appropriate broadcast or multicast address. This first level of checking is usually implemented in hardware.
Then it will need to determine what type of Ethernet frame it is dealing with (for historical reasons there are several).
Then it will need to determine which network layer protocol to pass the frame to. For an "Ethernet II" frame (the most common type) this is determined by the "ethertype" field.
Assuming our frame represents an IP packet it will then need to determine whether it is destined for an IP address that belongs to the local machine.
Assuming it is destined for the local machine then it will need to look at the protocol number and pass it up to the UDP implementation.
Finally it will need to compare the IP addresses and ports against it's table of sockets to determine which socket to pass the datagram to.
The number of possible transport protocols under IP is limited by the size of the protocol field. Similarly the number of possible network layer protocols for an "Ethernet II" frame is limited by the range of usable Ethertype values.
Furthermore using new transport protocols isn't a good way to get around port count limitations. Protocols other than TCP/UDP/ICMP will not be properly understood by many firewalls and NATs.