I was reading a textbook which says the need of length header for UDP:
The length field specifies the number of bytes in the UDP segment (header plus data). An explicit length value is needed since the size of the data field may differ from one UDP segment to the next.
So my question is:
Is the length header used to indicate the end of a segment?
And if the answer is yes, then why can't we use a special bit pattern to indicate the end of a segment?
It's just the way UDP was originally specified.
It could work without a length field in the header as long as the lower-layer protocol(s) made it possible to determine the length of a packet. That's why TCP is able to work without a built-in length field. UDP could have been designed that way; it just wasn't.
Is the length header used to indicate the end of a segment?
"segment" is the wrong term with UDP since it suggests that it is part of something bigger which gets eventually reassembled. UDP has the concept of a datagram or a message and such datagrams should be considered separate entities. This means that they can be lost, duplicated or reordered and UDP does not define what happens in this case. Dealing with this is up to the application.
... then why can't we use a special bit pattern to indicate the end of a segment?
This would be way more complex and time consuming to process than just prefixing the data with its length. It would mean that one would need to introduce a mechanism to either escape data which accidentally match the bit pattern or encode all the data in something like base64 (which is a waste of space compared to pure binary). It would also mean to actively scan the whole payload for the bit pattern and rewrite everything which was escaped or (in case of base64) to decode the data again.
Another aspect that's not been mentioned is 32-bit alignment of the data field: with the other three 16-bit header fields, the UDP header wouldn't have been a multiple of 32-bit words which the IETF aims for to speed up processing on 32-bit systems. The (actually redundant) length field was likely inserted to fill the otherwise reserved 16 bit with something useful.
Note the difference to TCP which doesn't sport a length header field, likely for the exact same reason. (With TCP, the segment length is derived from the IP header length field which UDP could have used as well just the same.)
