You have a way with words, Sally
It may just be because I'm a man, but women are just more physically beautiful IMO. Men have rougher features and aren't soft have scratchy hair etc. (of course men are 'valuable' too ... but in terms of physical beauty, I think nature handed the edge to women, then again that may also be a product of social conditioning)
But yes, if you compare men and women physically, there aren't a lot of differences except in how things are "laid out".
But let's assume that everyone saw themself with sexual features as being the same as how others saw themselves too - we could all be men or women or appear to outself as something that may not be specifically male or female (It just seems like that could be true, but I'm not certain exactly how).
Anyway, if that was the case, then how could it be that other people appeared as two different sexes? It would immediately seem that someone should either see themselves as the same sex as everyone else (which would really just be describing the property of being human and not specifically male or female, so in a sense there is a single sex - human), or alternately see everyone else as the opposite sex (the "other"), but that's not the case.
The only way it would seem possible that my suggestion could be true would be if we lined everyone up and at each step between people the sex was altered - same, different, same, different, etc.
Now here's an interesting thought - imagine that a child never leaves the womb but grows inside a parent - in this case the immediate "outside" for ones environment would be "inside" the other and we have one of the most obvious differences in terms of sexual organs, though the general "map" between male male and female sexual organs is not actually very different, just reshaped or rearranged - the further we moved away from this, the more similarly things could appear to be.
Anyway, it's not easy to figure out how such could be the case, but there still appear to be reasons why this could be true. (It'll just be another one of things that sticks in the back of my mind waiting for the pieces to fit right).