Where they are placed matters, but not in the way you think it does.

It goes from Move 1 to Move 4.

For example:
3,1,x
4,2,x
x,x,x

and

x,4,1
x,x,2
x,x,3

Both Move 1s go, then both Move 2s, then both Move 3s, then both Move 4s. Your Neji has lower Initiative than the other Neji, and your Guy has lower initiative than their Move 2. That's why your ninja are slower.