I am creating a custom protocol over UDP and I want to use a low number of bits for the sequence number. This means that the number can wrap relatively fast. Is setting the TTL for the outgoing packets to a low value a way to ensure that any packet received was actually sent less than TTL time ago?
No. The IP packet TTL effectively counts down the L3 or router hops the packet takes. Setting the TTL to a custom value n ensures that the packet cannot take more than n hops on its path. If it does it is discarded by the router on the nth hop.
TTL started out as a combined time/hop counter (hence its name) but quickly evolved to becoming a hop counter only.
To accomplish what you're asking you'd need to use a larger sequence counter or include an actual timestamp. Actually, using a 32-bit sequence counter covers extensive ground while taking up just four bytes in the datagram. Depending on the packet frequency, even 16 bit might suffice.