Of course you are not doomed! Or, at least, you don’t have to be. It sounds like there might be some miscommunication about each of your sexual expectations happening. Do you guys ever talk about it?
It could just be the way you are phrasing it, but maybe he’s lost sight of how exciting sex (between just the two of you) can be. It might be that you need to spend some time reminding him of how awesome your sex life is. Revisit oldie but goodie favorites that you both engaged in as a twosome, without any extras. This is not a one-off situation, however. Please be advised that you may have to attempt this several times to really get your point across.
Alternately, I would like to pose this question to you: do you still enjoy your alternate sex life? Is it a problem or is something else bothering you? Maybe you and your partner need to compromise how much time you spend having kinky and vanilla sex. No one’s sex life or desires are static; what was working for you may now not be as desired. That doesn't mean you won’t want it again. What I am trying to say is that sex, sexual satisfaction, sexual desires must be ongoing conversations that are revisited from time to time to ensure both partners are getting their respective needs met. It can also allow for new ideas and outlets to be explored, because of the more fluid conversation around it.
I want to float a hypothesis out, also. Just a thought, and I could be wrong, but maybe you are [also] seeking some affirmation from your partner that you still turn him on when it's just the two of you. Maybe you are just feeling like you are enough, and he isn’t giving you the reinforcement you need in the way you need it to feel reassured that you are still plenty for him. Sounds deep, right? All I am saying is that you want him to say a certain something, or do a certain something, that would make you feel all better about this – it would make you feel like he is still attracted to you and wants you, with or without all of the extra bells and whistles. What is that thing you want him to say or do? Now, does he know what that thing is? If not, it might help if you share it with him and give him the chance to fulfill it. Let him know what this means to you, so he knows how important this thing is and how much it will help the two of you going forward.