This is a dilution not abandonment issue.

Courts will look at the level of systematic tolerance. If you have a history of vigorous enforcement, it will be harder to argue in the future that a borderline dilution should be allowed.

If you allow borderline dilution, the court is going to consider what you have let other people get away with in the past.

It’s a bit of a catch 22