Has any court ever ruled that a trademark was abandoned, merely on the grounds that its owners didn't try to prosecute a borderline infringement case?

This is a dilution not abandonment issue.

Courts will look at the level of systematic tolerance. If you have a history of vigorous enforcement, it will be harder to argue in the future that a borderline dilution should be allowed.

If you allow borderline dilution, the court is going to consider what you have let other people get away with in the past.

It’s a bit of a catch 22