Distance to server is the most important factor.
Ironically, there are people that report low ping as a major disadvantage. My experience has been the opposite but everyone has an opinion on it. I sympathize with you if you always have high ping. If I connect to a distant server I get high ping too and it feels awful.