I think to be a sword user, the following qualifications needs to be met.
Have a physical bladed weapon.
Said weapon needs to be bigger than a standard kunai
Said weapon needs to be a primary method of attack
Said weapon needs to be primarily used in melee combat (i.e. projectile weapon don't count)

Given that, the only one I don't quite get is the orochi traitor.

As for calling them sword... well, sword was a bit literal in term of translation. The original wording is more like bladed weapon. Take seven swordsmen of the mist, for example, there is the Kabutowari. That's not really... a sword. There lacks, however, adequate words for the classification required. (Also, kenjutsu is not limited to "sword" exactly)

Which would fall under the "as a primary method of attack" bit.
But then comes the question of, just what amount of usage qualifies one on that regard. Sure, he doesn't use it a heck lot, but neither does sasuke or sai, as far as story goes. And if we are talking about in-game attacks, GNW sai and kimimaro does not use their swords to attack in game... I think...