I feel its not that bad an abuse of notation as kinda consistent with other areas of mathematics -

A coset, quotients r + I, affine subspaces v + W, etc. Not literal sets but comparing some representative with a class label, and the `=, +` is defined not just on the actual objects but on some other structure used to make some comparison too.

No. Equality is defined on sets, with exactly one exception, this stupid fucking notation.