In my opinion, I think that any animal will be calmer and more comfortable if they are comfortable with people and the daily routine. I've been working with wild animals for 15-16 years and unless you need to rehabilitate and reintroduce them to the wild, the theory is the same. It's always better for an animal that is going to be forced to live in captivity, whether it's an owl, a monkey, or a chameleon, to feel as comfortable as possible with humans, our husbandry routine, and mild handling for health purposes. A chameleon that is scared out of it's mind of people will be much more miserable in a cage in your home than a chameleon that is comfortable at least with the sight of you.
This doesn't mean pull them out of their cage and force them to interact with you for hours every day, but work with them slowly and patiently to build up trust - start with hand-feeding to associate yourself with food. Then perhaps move on to bribing them onto your arm. Then handle them for short, rare stints of time and build up.
Even if you never get them to be comfortable with handling, having a chameleon that doesn't turn pitch-black, hiss furiously, and fall all over itself out of terror whenever you open the cage is already a small victory. And he'll be happier for it, since it's inevitable that you'll need to clean the cage, mess with the misting nozzles, pick up dead feeders, etc.
I've gotten even my most aggressive chameleons to relax to the point where they would tolerate short stints of handling, which made a HUGE difference in our lives when they needed to be taken to the vet, treated with eye ointment, weighed to monitor health, etc.