Most members will tell you to handle him only when it is needed. When checking him over for any ailments, or cleaning his cage, or taking him out for some sun. Other then that, handling should be kept to a minimum.
Now, each chameleon is different. IN GENERAL, chameleons do not take kindly to handling, it stresses most of them out, some more severely then others.
It is rare to find a chameleon that ENJOYS handling, let alone even tolerating it.
Before you believe that you have one of those rare chams, you also need to realize that a chameleon may be prone to dart out of his enclosure and onto you hand due to there being something wrong with the actual enclosure, ex. too hot, or too cold, or even too small.
With all that being said, my personal advice would be to handle your cham only when you need to, unless you absolutely know beyond a doubt that he enjoys it, otherwise you may be doing more harm, then good.