Distance is just one of the factors, because your latency to the destination is largely from physical distance in many cases. Not always the case though, you could be taking some extremely sub-optimal path to get there. ISPs' Internet routing is generally least-cost (monitarily for them), not necessarily best path.