Removing for a tank or healer would be optimal not just for protecting your swordsmen, but also for utilizing the max swordsmen buff. A healer would be able to dispel any debuffs which could be really helpful, especially if you're looking for combos.