A lot of people, including me, are worried about the punching of Fascists, real or imagined, because it sets a bad precedent. But I'm also worried about the arguments being offered for this point of view.

Here's how it goes. They say that "we" have a fragile norm against punching people for political reasons, that engaging in street violence threatens to shatter this norm, and that this could get very bad.

Here's what they (and I used to) leave out: 

Under current norms, some groups of people already get to attack others with near impunity, and other groups expect to be regularly hassled, and to have to go out of their way to perform submission and harmlessness to avoid getting shot by the first group. This isn't a symmetric situation, and modeling it as such amounts to unprincipled apologetics for state violence.

Instead, model the situation as one of group conflict, in which there are currently power imbalances reflected in asymmetric norms about who can do violence to whom.

I still expect that a lot of street brawling with politically objectionable people will turn out to be ill-advised, but that's not an excuse for pretending away the preexisting violence in the system.

Related: Hierarchy and Wings, Nightmare of the Perfectly Principled, Should Effective Altruism be at war with North Korea?