The Sad Truth is that chams can get that way, especially Veileds. They are anti-social most of there lives, only coming in contact with each other over territory disputes or mating. Other than that they are loners, and like to be left alone. Another issue is the glass aquarium if he is in one. If he sees his reflection he will become either Aggressive or Passive. Either way you should get it out of an aquarium and into a screened cage as soon as possible.
Because he is a chameleon. Plain and simple. In cham-speak he is telling you to leave him alone. Have you noticed he always raises a front leg slightly when he is disturbed? More cham-speak, more defensive posturing.
When a dog growls & the hair along his neck and back raises, you don't attempt to pet. When a cat hisses and fluffs up his coat, you move your hand away. When a horse's ears are laid back upon his head, his nostrils dialate, he opens his mouth to expose his incisors and stomps the ground nervously, you back away.
Same thing with a cham...except he has no hair to raise, no incisors to show, no hooves to stomp the ground..LOL.
Handle him only when necessary. He'll be much happier then.
He also may be unhappy about his environment. He may be lacking something he needs, he may be ill. More information about your cham and husbandry might help as well.