Feelings about going to church.

Hi,

I’m married to unbeliever and we have 3 kids. Often I go to church alone because my wife doesn’t believe but she does go sometimes to support my beliefs. Still I feel socially awkward when alone at church and feel our relationship is being gossiped about. We are only couple like this in church and what should I do here ?