To answer the question of 'when is too much, too much?' the answer depends on the chameleon, but if you have a chameleon that doesn't mind being handled (eats while handled, is active and moving while being handled, explores while in your presence) then your chameleon could very well live on your shoulder for its whole life.
If your cham stays still, turns dark, tries to avoid being seen by you, and is inactive until you look away so it can make an escape, then I'd say just handle your chameleon when neccessary.
Some chams won't take to handling, some chams will. You can increase your odds by hand feeding a very young chameleon so they associate you with food, and then work on handling in in 10 minute increments every few days from there.