At this point, other than group related stuff, the population of your own server doesn't really do much other than for chatting.
Fox can be finished in 5-10 minute after it stabilizes, regardless of population
Sage, Masuri, Arena are Xserver.
Given that, both staying in whatever server you are vested in and going to older server are mostly ok.
Staying is kinda boring, might have nothing to plunder but you might get more reward in GNW
joining older server mostly will make you a bit useless in GNW, get plundered instead of doing them and not rank as high in fox.
So frankly, I'd say the difference is not nearly as big as it once was, due to Xserver sage.
(note, if you are not on the oldest server cluster, moving to a different cluster might have some impact on game play. Thou unless you are near top of your cluster, it probably doesn't.)