Do the dailies for Elite Instances. It gives you 4 every time. So far, that's the only way.

I shouild've clarified that the only guaranteed way to get the fragments is by doing the Elite Instance daily. The others are purely by chance.