Hi I'm a newbie in networking, I just want to know how IP datagram know its upper layer protocol is TCP or UDP? I know there is protocol field inside the IP datagram header that can tell the layer protocol, but how this field is set by transport layer?There is no such field in transport layer that indicators which protocol it is using?
When the application program communicates with the operating system, it usually says something along the lines of "open a TCP socket to address A port P"; for UDP it typically says "open a UDP socket" and sends A and P with each data packet. A given operating system might also offer "raw sockets" where the application program fills in the whole IP header, useful for specialist purposes including developing new protocols.
The operating system is then responsible for filling in the IP header and handing it off to whatever lower level is appropriate. In general, each level's header contains a code for only the next level up. For example, a particular ethernet frame might say its contents are IP, the IP header says its content is UDP, the UDP header says the contents have particular port numbers.
Operating systems vary in their interfaces, but the most common is the Berkeley socket interface, where the application asks for a "stream" (TCP) or a "datagram" (UDP) socket. Any detailed question about that should really go in another forum.