I'm a 6'2" guy and people only mess with me if they're intoxicated. I'm always completely aware of my surroundings unless I'm in a crowded place that should be safe (like malls and airports).
All my sketchy encounters were on public transit, so I no longer take it in my city (transit in Denver is THE FUCKING WORST). I've been threatened and followed multiple times by drugged out weirdos, but let me tell you about the two worst incidents: 1) I was jumped and attacked by a random guy at an empty train station, there was nobody around to help or even witness and I ran for my life as he chased me; 2) guy at a bus stop recognized me from my workplace and he followed me on the bus to my girlfriend's place and then on another bus back to my home and he started hanging around outside my workplace hoping to run into me. So yeah most of the time I'm hyper-aware of everything happening around me in public and you're right to be as well.