I've always thought this: If you found yourself in a cage, unable to escape, and had this big terrifying creature walk in front of the cage every day, capable of opening the cage, and terrified that it would hurt you somehow, you would live with a certain level of stress every day. Where as a chameleon that isn't scared of you sees you every day, associates you with food and maybe outside time, isn't going to see you as that scary thing that could hurt him at any moment. I believe the second chameleon will live with a lot less stress over his lifetime.
I have mine out several times a week, usually to move them inside and out to get sunlight, but my panthers are all pretty comfortable with me. In fact, they come up to me in case I have food.
If you can work slowly to build trust with your chameleon I believe that they will be better for it in the long run. Even if you never get to hold them, if they are comfortable with you in their cage, that's already a great step forward.
