Brightly colored does not necessarily equate to "happy". He is definitely going to become more brightly colored, or fired up, when he's in sight of a female because he is trying to court her for mating. He's trying to show off. Any male in sight of a female will become more colorful. So the fact that he was always more brightly colored while sharing a cage with the female probably means he was constantly trying to mate with her. That is way too stressful for a female and will cause her chronic stress leading to poor health, if not outright aggression to try to keep him away. So he is not actually lonely, he is just not showing off all the time. And he shouldn't be so that's okay.
They can also become more brightly colored when they are afraid or stressed. My veiled is never prettier than when I have him out to take his picture - he hates being handled and he is afraid of the camera! So in that case brightly colored means he is anything but happy. In your cham's case he might have even been intimidated by the female and being more brightly colored was his stress response to her.
What you're seeing is his normal coloration while he's relaxed. He does not need a friend.