Kay maybe its a Yoda thing lol. I think honestly I just gave him to much space. I think it made him fear me more. Instead of introducing myself to him early on so to speak. I held him again tonight for a few minutes and after I put him back he came out from all his plants hung out on his branches, moved around a bit, ate some in front of me, hung out some more. I think I might be growing on him lol. Tomorrow I need to clean his enclosure good, not just a clean up and he might be a little mad after that

but it needs to be done. I clean up his cage every day to every other day then once a week or so I clean to really good. Hopefully he gets used to it. Hes healthy, active, eating well, and drinking well so I have nothing to worry about right now.
Try thinking from your chameleon's point of view. You are a big chameleon-eating monster. That's how he views you, and probably will for the rest of his life. I am sure others have chameleons who truly are tame, but if you treat him as if he is afraid you will eat him at any time, you will slow down your movements and be more aware of the stresses you cause him, and ultimately make him more tame. Think of how you would feel if one of the monsters from Aliens reached into your living room while you were watching TV. That is his perception of you. He will not get "mad" at your cleaning his cage--he will become afraid, very afraid.
What you experienced when he seemed less fearful after you handled him is that he was temporarily desensitized to your presence after you flooded him--you forced him to submit to a very fearful experience with no opportunity to escape. That kind of desensitizing doesn't always work and has a habit of backfiring, with the fear returning at greater intensity. You can actually teach him to be more fearful, and cement that fear into his little psyche.
Young chameleons often appear more compliant to being handled than adults, but that complaisant behavior as babies doesn't mean they will be that way as adults. They might be just as afraid as the big hissing, gaping veiled male that the baby might very well grow up to be. They just may not have learned that survival strategy yet. Or that survival strategy--aggression--is not the best for a baby chameleon. While they might tolerate handling, they still might be terrified.
Yes, they really do get used to humans. I have a group of wild caughts and all have settled down. Two were suicidal if handled, choosing to leap into space rather than risk being near me. One was almost impossible to find in her heavily furnished enclosure, diving for cover if I entered the room. I only knew she was alive because I hadn't found her body. She took a roach out of my hand the other day. Several will sit calmly on my hand, but I know they really are terrified, just not as much. I just don't think it is fair to my animals to torture them with handling.
A lot of times, what a person describes as happening in an interaction with a chameleon is not at all what the chameleon experiences. There is no way anyone can know the inner workings of a chameleon's mind. When I hear, "happy to be handled" or "he loves to sit on my shoulder and--this one really got me--snuggle," I suspect the person is not being observant and has not done any research into the natural history of chameleons.
Running over to a door to greet you does not mean the animal has any emotional relationship with you. He might want to escape the cage or expect food. They do learn and will interact with you even when afraid. An interesting taming method used with wild-caught adult parrots was to use the trainer's retreat as a reward for the parrot allowing the trainer to come closer. Sounds counter intuitive, but the parrot learned they could make the trainer go away if they tolerated the trainer close for a moment. The parrot allowing the trainer close proximity did not mean the parrot liked the trainer or feared the trainer less, it just gave the parrot control. A survival instinct for chameleons is to be still, so sitting quietly on your hand does not mean they are happy. It might mean they are in mortal terror waiting to be eaten.
Don't underestimate the damage done physiologically to an animal exposed to repeated near-death experiences. The cascade of stress hormones does many things, one being suppressing the immune system, which is why stressed animals get sick and die more often. It takes hours for the blood chemistry to return to normal after a stressful situation.
I hope what I've written is helpful and gives you some ideas on how to integrate this wonderful animal into your life.