The PATing device uses a combination of IP address and port to create a map of unique entries, allowing it to understand what traffic belongs to which device.
The port allocation needs to be able to account for multiple connections to the same service over the same port. This is critical for traffic that uses the same source and destination like NTP.
Consider if you have all three servers set to the same NIST NTP server. If PAT didn't assign a unique port to each flow, you'd end up with:
192.168.0.1:123 ->PAT->x.x.x.x:123 -> 129.6.15.28:123
192.168.0.2:123 ->PAT->x.x.x.x:123 -> 129.6.15.28:123
192.168.0.3:123 ->PAT->x.x.x.x:123 -> 129.6.15.28:123
(where x.x.x.x is your public IP being used for PAT)
The return traffic from NIST would just be 129.6.15.28:123 -> x.x.x.x:123 for all three flows. How does the PAT device know, based purely off of source/destination IP's and ports, which internal server the return traffic goes to?
By assigning a unique source port during PAT, you end up with something like the following:
192.168.0.1:123 ->PAT->x.x.x.x:123 -> 129.6.15.28:123
192.168.0.2:123 ->PAT->x.x.x.x:124 -> 129.6.15.28:123
192.168.0.3:123 ->PAT->x.x.x.x:125 -> 129.6.15.28:123
Now the return traffic will be destined to a unique port, which the PAT device can untranslate back to the proper source.
This isn't just an issue for 'symmetric' services like NTP. It is entirely feasible that you have multiple clients connecting to the same external server, and eventually two or more collide on their choice of source port.
Also to Ron's point in his comment, this is all very general and the exact implementation depends on the device and the specific type of NAT being employed.