The presence of ann invariant implies invertibility! That doesn’t seem true.

Right, e.g. projection onto a subspace of a vector space leaves that subspace invariant. I suppose that the existence of an inverse is axiomatic (and maybe up for debate for whether it is required to consider something a "symmetry"), but assuming one exists, then it leaves the object invariant. You could also observe that it has to be invertible on the object, but not obviously necessarily the ambient space I suppose. Representation theory perhaps has a satisfying answer to that part.