As a general rule it's better not to mess with them. I've had a few that really couldn't have liked me less, and I tried to leave them alone as much as possible. I would take them to the outside cage for sun and that was it, not a lot of handling. But I've had others that were much "friendlier" so those will get taken out often. It depends on their personality, some are way more tolerant that others.
However, I will say that even if you can get an aggressive chameleon to hand feed, I think they'll be happier for it. If you were terrified of even the sight of humans (like one or two I've had), think of what it must be like living in a home where you're trapped in a little cage, and the big scary thing can open and enter your cage at will. It would be like living in a shark diving cage, and the sharks have the key! But if you can at least make them comfortable with the sight of you I think it makes for a more tranquil animal, since they at least tolerate you as a normal part of their every day. So, in my opinion, if you can get them to at least hand-feed comfortably I think that's a good thing.
I think the main thing is for them to understand that you are not a threat.
In the wild, they live in harmony with animals much larger them themselves and know they are not preditors, so a human should not be much different.
They are rather defensless creatures and need to be super cautious to stay alive.
My male veiled is just now starting to 'communicate" his wants to me.
When he sees me, he will walk close to the front of the cage, stair at me a bit, then walk over to his cricket feeder and stair at that. Sometimes he even climbs inside the feeder.
So then I say to him; oh Smeagol, I get the idea, you want some food.
When I bring the tube of cricks to him, he comes running from whereever he is as soon as he sees it in my hand.
He barely gives me a chance to dump the cricks into the feeder!! like he is starving to death!!
I think that is a form of communication.
He also will no longer hesitate to eat from my hand, even if it means having
to walk onto my finger to reach.
He does not want anything to do with being held though.
I approch him from the front and try to slide my fingers under him and slowly lift.
He just slowly turns and walks away, keeping an eye on me all the time!! LOL