When I was late teens and early twenties, I kept getting propositioned by gay guys. It started to bother me - I've never been in any way homophobic, but I've also never been attracted to men - and I started to worry that there was some vibe I gave off that made people think I was gay.
Then I mentioned it to a gay guy who I knew, and he said that the vibe I give off is of someone who isn't judgemental and who wouldn't react badly to being asked. Maybe there's something similar or analogous to your situation.
Edit: propositioned, not preposition