This is a good food for thought. Semi-related: It reminds me of the fact that smartphone cameras nowadays use ML based filters to make faces prettier before saving in the first place.

Perhaps we can use clever experiment design to address it, like doing a "full-reference" evaluation (e.g., pick most similar two images, out of encoder A, encoder B, and original) rather than strictly A/B.