Dude, it's been a pain to do Arena ever since it became cross-server, you're not the only one experiencing this kind of situation. I too, have experienced fighting people powerful than me.

Cross-Server sounds good in theory, but the matching, just like in Matsuri and SWB, are....dumb.