It's not an issue with the servers.

I would strongly advise on refreshing during a cross-server event, As quitting can and will leave you from re-joining.

Because of the packet loss to your opponent, you've intentionally stalled the match by existing. It's quicker to just simply * it out.

Your opponent would have to wait till the connection stabilizes for a return to combat.

So basically, just avoid refreshing in Sage. Especially during battle.