By physically opposing fascism, I assume you mean they are taking specific practical actions rather than becoming one with the platonic concept of opposition to fascism.

It may seem an obvious or insignificant point, but it is critical here. If they physically oppose fascism by following and filming ICE, I'm very much on board. If they oppose it by molotoving innocent local government buildings, I am against. If both of these actions are the concept of opposing fascism, what does it mean to be against that?

Antifa are belligerants. They undermine protests by having the maturity to die for a cause but not to live for one. One can be against that without being fascist.