You seem to have it backwards. Some transport protocols, e.g. TCP and UDP, use addresses that are better knows as ports. Other transport protocols use other or no addressing. Both TCP and UDP use the same number range for their ports, but they are not the same ports. For example, TCP port 12345 is not UDP port 12345. You cannot, given only a port number, know which protocol is meant.
The Protocol number field in the IPv4 packet header, or the Next Header field for the IPv6 packet header (same as the IPv4 Protocol number), will tell you which protocol is being used, and it may be a protocol other than TCP or UDP. IANA maintains the Protocol Numbers page that explains which protocol number is which protocol.
Look at this from a perspective of the layers in the network stack. Each lower layer encapsulates the datagram of the higher layer as the payload of its datagram. There will be a field in the header of the protocol at a layer that tells it to which process it should send its payload.
For example, ethernet can carry many different protocols as a payload, and it doesn't know or care about the protocols, so how does it give the IPv4 packet in the payload to the IPv4 process and not the IPX, IPv6, etc. process? The ethernet header has the Ether Type field and if the field contains
0x0800, then it sends the frame payload to the process registered for that number (IPv4), or if the number is
0x86DD, then it sends the frame payload to the process for that number (IPv6). IANA maintains the IEEE 802 Numbers with the Ether Type for the IEEE protocols (ethernet, Wi-Fi, token ring, etc.).
As I explained in the first part of the answer, IPv4 and IPv6 use the protocol numbers maintained by IANA to determine the process to which they should send the packet payload.
TCP and UDP use addressing commonly called ports. Applications register with TCP or UDP for a port number for the protocol (TCP or UDP) that the application wants to use. Any UDP datagram or TCP segment coming in will have the destination port number to which the protocol will look to see which process has registered with it for that port number, and it will pass the payload to that process, or drop it if no process is registered for that port number.
There are services that have registered with IANA (see Service Name and Transport Protocol Port Number Registry for well-known port numbers. Unfortunately, that does not keep an application from using a port number that is well-known for a service to be used as a different service if the port has not been claimed. For example, a web server normally registers with TCP for port 80 for HTTP and port 443 for HTTPS. If a host is not running a web server that has registered those ports, any application is free to request use of those ports. Sometimes people will register SSH with port 80 because a firewall has blocked port 22 (well-known port for SSH), but the firewall allows port 80 traffic. Some people, for various reasons, will set up a web server on a different port than the well-known HTTP port. That forces anyone trying to browse that web server to explicitly enumerate the port number in a URI.