Your movements need unhindered flexibility, and wearing armor would just get in the way.
When you are not wearing any armor, you can add your Soul to your AC.