I love perfume, for myself. Not trying to be sexy with it, but like to smell affirmatively good not just clean. I don't like "clean smelling" perfumes at all, the laundry musks are not my thing. Some of my kids like those, I don't. Guys, as long as they don't smell bad, that's a win.
But I have twice in my life been ridiculously, outrageously sexually attracted to guys I didn't like the look of, and could only conclude my body must have thought they were biologically compatible. It wasn't an obvious smell or anything, not consciously, but I do think it was a smell that I could subconsciously perceive.
It's a sort of inhumanly clean sorta soap smell in some perfumes, usually in the understated minimalist ones. I like florals, fruity stuff, dirty musks, spices, but apply lightly because it's for me, don't want much projection. I don't think guys usually care. I remember once I asked 3 men and one said "only vanilla", one said "anything but vanilla" and one said "I don't care"
So I went on just pleasing myself. Anybody close enough to smell it hopefully is already into me, lol.
Oh! I also wanted to note, it took a long time for my husband to believe that I was wearing perfume. He would say "you smell so good" and I'd say "yeah, Mure et Musc Extreme" (or whatever I had on), and he'd say "no, it's a little bit your shampoo or body wash mostly just you, not perfume" but it was definitely the perfume.
So it is possible that some of the scent you think is just person is actually perfume, a lot of them wear down to really pretty, very human smelling skin scents.