Well general body shape and height are pretty easy tells.

It is not an "excuse." If I'm attacked by a man at night, I will have a more difficult time protecting myself than if a man was attacked at night, on average. Not to mention that you have to ask yourself, what is the attacker after? Because being sexually assaulted or raped is so personal and awful that it's natural to have a higher amount of fear of that outcome.

Like how if you were equally likely of being jumped at night by a stranger as you were of being abducted and having spiky metal spears shoved in your urethra... you might recognize that's it's not super likely you'll get jumped, but you would be way less likely to risk it since the fact that sounding is just too awful to chance (for the vast majority of the population).

Also... my sample size is too small to be statistically relevant, but 100% of the men I know are sure they could "take" a random female attacker.

Unrelated; asking that question was difficult but not for the reason I expected. It was like a game! "Do I have super powers in this scenario?" "Is she armed? Am I armed?" "How much time do I have to prepare?" And my personal favorite, from a particularly well-traveled friend- "Is she like a random person, or one of my exes?"

I can easily see if it's a women or a man just based on how they move from a distance.

I guess it depends on how dark it is. If it's pitch black, nobody can see, but usually you have some shape you can see.