You said..."handling just occasionally cuts off a big part of their lifespan"...
First off, how can anybody measure that? How do you know how much their life is shortened or lengthened by diet or temperatures or a lot of other things in their lives?
I'm not saying that handling some chameleons might not cause them a lot of stress and even lead to health issues, but it's a very individual thing...just like in people, some handle stress and uncomfortable situations better than others.
This doesn't mean I'm saying go for it ...handle them as much as you want either.
I try to judge how the chameleon reacts and let it make part of the choice....so if I have one that comes out on my hand readily, I let him. If I see it's stressing him out then I don't handle it any more than needed. You will even find that one species will react to you differently than another.
Then there are WC's that can't adapt to a captive environment at all and you have to be very careful with. Those ones sometimes have a very hard time with handling and can die sooner from it IMHO.
I think we have to learn to "read" the chameleon sometimes. Just my 2 cents worth.