This has been troubling me too since I started playing!

Sometimes the game allows you 2-3 seconds to think before the opponent casts his skill, but other times even a ninja in 4th position might cast it faster than your #1 ninja!

Unfortunately no one bothered giving an explanation so far, so we try to play the guessing game and hope for the best!