Hey there !

Charms scale off cave stones, you need them to get any bonuses from charms. In other words, you don't have any key stones so that is why you are not getting any power from charms.