Totally normal. Chameleons change color for a number of reasons, including feeling threatened. How long have you had him?
My panther was quite frightened of me around that age, but he got more tolerant with time. Make sure you're patient and don't force him to be handled. Just work on hand feeding and gaining trust.