Sometimes acknowledging that a thing is distracting is the best way to let it stop distracting you. The word is distracting to you, but you have a choice about whether it remains distracting.
I see these things as a pendulum. And as a sense of perspective.
On the pendulum front it is that in order for things to truly get to a place of equality that means the middle of the bell curve needs to be at 50/50. Which to me means that the easiest way to approximate that is that for every space that is purely patriarchal there needs to be a space that is purely matriarchal and similar proportions of spaces at levels of skew less intense but not 50/50. Until we get that the society is patriarchal. I think that when I've seen this achieved in micro scale it is when the pendulum of power at a local level shifts from one side to the other back and forth until it lands in the middle. We have a society that is so patriarchal that I don't think it will ever be allowed to swing far enough towards feminism that there can be a correction towards the middle. And so in that context I think that something that is "against patriarchy" being called "feminism" to mean the eventual goal of being equal makes sense.
This brings me to perspective. As a white person I've been working very hard to understand what it means to be white and how that has impacted my perspective. I only really see how toxic and skewed my perspective is on this front when I hang out with several people of color at the same time or with a person of color who feels safe enough to call me all my bullshit or call society on its bullshit in my presence. Many of their complaints can be felt on a few levels. On the first level a white person who is stuck in that perspective can either say "Of course but this should be true for everyone." or they can say "I don't see this as a big deal." When I am able to delve into the level of toxicity of how the power ascribed to whiteness has seeped down to the core of this society I can actually feel the harm that the latter seemingly minor things cause and I can understand how truly ridiculous it is the contradictions involved in the white discourse among the major things. Unfortunately I still find it incredibly difficult to express that understanding and even to hold onto it as I leave the presence of my friends of color. One training I went to called it the water we all swim in. And we swim in so much sludge and have for so long that we don't even see it.
Side joke so that makes sense: An old fish swims by some younger fish and says "how's the water today?" and the younger fish ask "what's water?" Sometimes things are so infused that we can't even see them without real effort.
I feel the same thing about feminism. The patriarchy is the same way and it is very difficult to see it at the deeper level. The things you expressed for your views that make you a feminist could be at one of two levels. If they are at the obvious level it is likely that you can't understand how anyone feels differently and why this is even still a conversation we need to be having. If it were at a deeper level I think the word feminism would make more sense.
I believe there is a combat. Not between individuals but between ideologies. It happens along the gender line and it happens along the race line and it happens along the socio-economic line. The socio-economic one is the one that I feel has done the best job of stating the fight as I've seen it. "They only call it class warfare when we fight back." This can be applied to all the power structures. When we call it "race wars" it is when people of color get pushed by the bullshit to the point where they feel there is no other option. I think that feminism is the same way and why you feel somewhat attacked.
I also believe that it is not a dichotomy but a spectrum. I see the toxic water kinda being like those airport sidewalks. And the direction they go is the defaults of power. So in the absence of other actions we head towards patriarchy, white power, and a strong believe that the people with money all earned what they have and the poor are lazy. (among other things) When we start realizing that we are on a moving platform in society and that we are heading the wrong direction the first step is to turn around. The next step is to notice that we are moving away from the goal. And the final step is to start moving in the right direction.
Moving in the right direction though may not be enough if the rate of the moving sidewalk is faster than the rate of walking. Not only might we not get there or even stay put we might still be moving in the wrong direction. It can be super frustrating to be judged by your negative progress on this moving sidewalk, especially when you are actively moving in the opposite direction. But there will be people that do that because unless we get people to not just turn around and start moving and start putting in real movement nothing will change.