I know for a fact that handling snakes when they clearly dont want to be held is basically mandatory, unless you want an animal that sees you as a threat and has the ability to "deal with it" (large pythons, anacondas)
It also helps most other species calm down as well and eventually see people as non threatening, or even interesting and fun, a state of mind they would never reach if we did what they wanted all the time.
Doubly so for varanids.
Granted these aren't chameleons, but a case could be made for basic conditioning, an animal that begrudgingly tolerates you is healthier in the long run, than an animal that fears for its life every time it sees you.
Ultimately it's balancing your needs with the species, a panther handles contact better than trioceros hoehnelii for example.