By all means find a new server to join so everyone is fresh but dont stress too much about it.

In the future many things will be cross server across ALL servers. Which is great because it allows new players to join old servers with friends and still be able to compete.