As Yujin has explained, you can only swap between skills of the same level. If you swap to a skill of a lower level/rarity you will not be able to go back up and if you used fragments to obtain it then you will subsequently lose those frags.

This is outlined on the the Skill Trial Interface so in future please be aware and careful when changing skills.