It'll come down to the temperament of your chameleon. If you have one that is obviously horrified by the very sight of you, leave them alone. But if you have one that reaches out to climb on you, then it's their choice and that is ok. The time will depend on how long they are OK to be out and about, and as soon as they seem restless or uncomfortable they should be put back.
I have had chameleons I rarely ever messed with and others that were out daily for hours, it just depended on their temperament.