No. Only temporary buffs get removed. You can lower someone's permanent buff with your own ninja's skill, if has one.