Yes, that's totally normal. You shouldn't have the cages positioned so he can see her, it will stress her out and frustrate him.
I get that it might be difficult when you get him out, but I put a big towel over the front of my females cage when I need to get my male out. He will always know where she is now (unless you move her), so if you keep him in the same room that will happen. There is no need for a prolonged handling session really, but if you are trying to get him used to you I would suggest taking him into another room for it.
