Depends how much they tolerate. With my boy, Daedalus, I left him alone a week or two and then started leaving his cage open so he could choose when and how to come out. Once out I'd put my hand so he'd have to walk over it to get to where he wanted. We built trust like that and now I can handle him as much or as little as I need to, including visits to the vet, appearances at a biology lecture or reptile show, or visits to the homes of friends. When he was little I probably handled him a few times a week - maybe not daily, but certainly often.
However, the chameleons I've had that have been aggressive or scared I have left largely alone.