Your luck isn't terrible. It's just that the chance for something is very low.

I noticed a lot of players get a Rare Ninja around 60 or 70 scrolls. However, some get in like 20 scrolls.

RNG is RNG. It's not something you can control in this case. (Unless you go for the 400 Scroll Ninja, but that's a different case)