I have felt how you felt. Things actually, with some misogynists, have gotten worse recently in my experience. What that means now there is no uncomfortable peace with these people, now they either require me completely folding or holding my chin to get (metaphorically) punched. I have chosen the later, my pride won't let me be walked over as much anymore. For good and bad.
I have learned to not try and placate. If I am doing that, why stand and have opinions at all? If I just let them talk me into a position that actually isn't my position or softening my position, was I really standing at all?
I also have started trying to be actually gruffer. More in charge and direct the situation. A lot of people don't know how to handle the confidence. For the misogynists against me? I have cut them off (boundaries for acquaintances who are jackoffs) and made sure to cya (cover my ass) like get email or text or something with their lack of engaging, then I have "they let petty opinions get between being professional" which usually businesses/HR will accept and put them in the wrong if all they did before was be disagreeable.
But really, try to drop the self-deprecation if you can. We pick it up to be cute as teens, but it really makes us fail in the workforce and it actually can get really old to be around for everyone. If anything, it makes them think they're right about you (whatever the weird opinion is). If you are a thoughtful person and you have formed your opinions with care, why should you back down at all? Heck, who are they to step to you? Why do THEY have the right to be jackasses just cause they don't like someone? You're the adult trying to do your job. They are being unprofessional!
It sucks your boss let that person undermine you. If you can, I would personally ask my boss to direct everything to me, and make the person either act out and get reprimanded or they get on board.
I wish you luck. I don't get along with as many people anymore, sometimes it sucks, but those who rise above the rest are more worth it.