I'm on the pro-handling camp, personally. If you can work slowly and with patience towards a chameleon that doesn't have an emotional upheaval every time you remove him from the cage, I think it'll be best in the long term (for cage cleanings, moves, vet visits, giving medicine, etc.)
You can start trying to hand feed now, if you want to try. I would start by trying to hand feed the first insect of the day, when he's most hungry and most motivated to get it. Keep very still and be patient.
And as far as handling, keep it short and sweet like it sounds like you've been doing. If he seems tolerant, great, and if not skip it and try again another day.