Your movements need unhindered flexibility, and wearing armor would just get in the way. When you are not wearing any armor, you can add your Soul to your AC.