Good advice above OP. I've been fortunate not to end up with any particularly spicy chams, even rescued wildcaughts. If any cham has an excuse to get physical with a keeper it's them

.
Sometimes when getting a new cham more used to me, I'd hold a branch in one hand and coax the cham to step on it using my other hand. Then
walk away from the cage while holding them. The cage is territory that must be defended. A cham will often lose much of its attitude when away from its territory. I might just sit somewhere quietly holding the stick/cham for a while. Maybe near a window so the cham can get a little distracted by the view and forget they're close to the thing they distrust. Spending time closer to me (but not being touched) helps it learn I am not dangerous...just a rather strange looking fixture in their day-to-day lives.
Remember, no cham enjoys being stared at...so while holding the cham I make a point NOT to stare directly at it the whole time. Just take a casual glance at it occasionally instead. For most animals, a direct stare is a threat. Chams know what's about to happen when THEY stare at something and they'll tend to interpret your stare the same way. Avert your gaze as much as you can.
Hand feeding helps too, as the cham learns you are a source of good things. Sometimes I've held a stick toward a cham in one hand and a favorite feeder in an open container in the other. I hold it far enough away that the cham will need to step onto the stick and move towards me in order to get the treat. I'll try this for a few minutes, but if the cham doesn't take the offer, I'll walk away and try again later. They don't have much of an attention span so it's pointless to persist. Its stressful too. More short sessions will work better than one long one.
As for feeling you aren't spending enough time caring for your cham, what's happened is that you've settled into a routine. You've figured out little efficiencies in how you do things. Established routines simply take less time to complete. Reminds me of new parents who just brought their first baby home. At first, they hover and flutter around anxiously, worrying and second guessing themselves about it every moment. As days & weeks go by, they do less and less of it.
One other thing to think about is what you actually are afraid of. Getting bitten? It's understandable, but none of the species typically kept in captivity can really hurt you. A bite is more startling than harmful even if a cham has been warning & bluffing you before ever following through. As they say, fear of the unknown is often worse than reality. When I first started keeping Melleri I wondered just how powerful their bite would be. They're big. All of mine were wild caught but only one was nasty. I was very anxious about handling it. Finally, I got fed up with worrying about it. The next time the cham struck at me I didn't pull away. I let it bite and kept still until it released me. It didn't chew, it let go. All that happened was a slight cut and bruised thumbnail. Not nearly as terrible as I thought. Once I'd gotten that over with the cham's bluster didn't bother me nearly as much. Kind of a "bite the bullet and let the bullet bite" approach.
After that, if one my chams blustered at me I viewed its attempt to intimidate me with more humor. I'd just chuckle at it (well, not directly at its face...no reason to humiliate the poor little guy) and found it endearing, not scary.