I think your main problem would not be the toxicity of the plant, but the other dangers that your chams face: predators (dogs, CATS!, birds), overheating (leaving young chams out in direct sunlight is a danger) and finally escape - chams move far quicker than most people imagine.
So if you take chams outside and don't put them inside an enclosure, make sure that there is somebody there to watch them at ALL times! (even in an enclosure the risk of overheating doesn't go away, so keep an eye on them)
If you have the option, why not build an outdoor enclosure filled with safe plants? It really is worth it - there is nothing better than regular natural sunshine for your chams.
I would never put them out there without being carefully watched & and only for an hour or two anyway. I just thought that my little hydrangea tree would have been a nice place for them during that time.
Thanks for the info.