A NAT router doesn't know when to remove a UDP mapping - it guesses.
The router simply ages (or times) out the entry when it hasn't been using for a period of time (a few minutes usually).
With TCP, there's also a similar aging/timeout to make sure that forgotten or lost sessions don't pile up but it's much longer.
UDP or TCP aging is a trade-off between router resources (fast aging) and compatibility with slow low-bandwidth sessions (slow aging). Sometimes the default settings require tweaking for your workload.