Major domains have servers around the world, and DNS set up so that you get an IP address that's close to you.
You can avoid this by pinging a specific host. For example, the Debian (GNU/Linux) project has a list of mirrors in countries world-wide. Most of those are specific hosts that will look up to the same IP regardless of where you are. Especially university software mirrors are almost never behind a content-delivery proxy network.
mirror.csclub.uwaterloo.ca is in Waterloo, Ontario, Canada, and does respond to ping requests. I get ~37ms ping time to it right now (early morning) from Halifax, NS, Canada.
mirror.aarnet.edu.au in Australia is probably one of the farthest away from me; Australia as a whole is network-wise far from most of the rest of the world. (although South-East Asia is closer than most to it) My ping time is ~295ms.
And yes, speed-of-light delays imposed by distance around the earth's circumference are a major part of that, as well as routing delays on hops. (And remember, it's speed of light in glass fiber not vacuum. The index of refraction of the core of an optical fiber is often something like 1.3 to 1.4, so speed of light is
c/1.4. (It has to be a higher index of refraction than the cladding to create total internal reflection, which is the whole point of optical fibers.) Modern fibers for use over long runs do try to keep their index of refraction down as low as possible for this reason.
tracepath to find out the network path your packets take.