I am a programmer and we can filter out the packets in the router to manipulate the IP or MPLS headers. My question is there any way to distinguish between UDP 1 byte traceroute data vs user traffic data? The traceroute udp packet will have different udp source and destination port number and there is no standard port for it. Is there any IP fields through which I can distinguish traceroute packet?
Traceroute relies on TTL to find each hop.
The first traceroute packet has a TTL of 1, then the value is incremented by 1 for each following node.
Regular traffic will (almost) always have the default TTL value from the operating system, usually 64 or 128, this value being of course decreased for each router traversed between the originating host and your packet sniffer.
So, unless you are performing a traceroute for a destination that is more than 30 hops away (which is larger than Internet diameter), a filtering on the TTL field should give you a good match.
There is an actual traceroute protocol, which has now been deprecated (see RFC 1393, Traceroute Using an IP Option). This may still be used by some OSes, but some OSes use ICMP, and some use other protocols, but the key is incrementing TTLs.
The key to look for is small TTLs in the IP packet header. The minimum TTL that most OSes start with is 64 (some start with 128). You can send packets around the world in about a dozen hops, so if you see single digit TTLs, you can assume that you see traceroute packets.
Traceroute is often misunderstood. It is a useful tool on you own network where you understand the network design and know what the route should be. Using traceroute on the Internet can be very misleading because some ISPs in the path may look for ICMP or traceroute, and then reroute those packets. That is to prevent you from casually discovering the internal network of the ISP. In that case, the path you normal traffic takes can be very different than what the traceroute shows you.