I've had several ferns, that were already potted plants (i.e., were adjusted to "captivity" fail to thrive in my pygmy terrarium. It was primarily a matter of the light level being rather higher than the ferns like (even a Reptisun 2.0 at so close a distance is more than a shade plant wants). If you can give them enough shade (dense overstory plantings), it might work, or you can give a fern a month in the cage, then replace it with another and put it in a shadier environment to recuperate. You might also try growing them from spores (the little dots, sometimes lines, on the underside of the fronds; some ferns, like the cinnamon fern have specialized, cinnamon-colored, spore-bearing structures). Scrape some of these of the plant onto moistened substrate. I've grown ferns in this manner for several months under fairly high light, and the exposure from the very beginning might acclimate them.