did you have them in a pod, when you say mold growing over food source? I would think as long as the ones you gave him were healthy and alive at the time you should be ok. Maybe someone will correct me if I am wrong.
Silkworms need to be kept clean. They have very poor immune systems and are prone to infection. They need good ventilation and to be kept fairly dry. Mold on the food will definately cause them to go downhill fast. I do not like raising them in pods. The food either dries out or molds when I leave them in there. I raise them in steralite tubs with a screened top, it seems to work quite well.
One common disease in them in Grasserie. For more info on its symptoms and prevention goto these links.
I like to clean the silks every other night. Remove any dead or questionable looking ones to a seperate container. Keep things as clean as possible and it should keep deaths to a minimum.
As far as your cham, He should be just fine. As long as the silks were living when you fed them I doubt there will be any harm to the cham. I have read about people feeding off silks that had grasserie with no ill effect FWIW.
How many did you have in there? I think that container is too small to have them live to their full sizes. Next time put them in a bigger container. A large critter keeper works for me. You have to keep it clean too. Poor things.