You are right, it shouldn't matter, legally but what you have to realize is that some jobs are simply more macho than others and attract that sort of people. I would imagine that a skinny computer geek type guy would probably be harassed a bit more than the average shipyard welder no matter how great he was at sticking metal together.
The other side of the coin is that some of those macho guys you work with may be distracted by having a beautiful woman working within eyesight with her head covered in a hood and only the shape of her butt to indicate her femininity. Management probably notices the slow down due to distraction and discussion about you.
In my opinion, even though they changed the laws so a woman can go for any job she pleases, it does not mean every person in the company is that liberal minded. I think it is ridiculous for a woman to even want to do such a job. To me it is the same as a man who chooses to be a bra salesman. Why?
I am a fair minded person, so here is your opportunity to discuss the matter with someone who feels the same as the managers but I can't fire you so tell me why you even chose such a "manly" job in the first place?