handling advise?

ive handled my cham a few times but sometimes whilst on me he will puff out his chin but only for a few mins and go dark.he never hisses or goes to bite or anything.
shall i back off from handling him or just carry on?

With the information you gave, going to have to make some guesses. I assume to handle him you reach into the cage and catch him. Try and feeding him for a while get him used to your hand. After he gets used to this, try to encourage them to walk onto your hand by holding food close to it. Good luck, and have patience.