> but people won't be able to see the artifacts

But that is correct and why any proper study should first train the participants on the type of artifacts and trade-offs. Like the reference to JXL choices: think its creator mentioned that AVIF might look nicer for faces because it was "smoothing" them but it was a worse representation of the original. So the study participants would need to be conscious of that trade off and be clear in terms of what they're asked to evaluate - whether the result looks nicer or whether it represents the original better.

This is a good food for thought. Semi-related: It reminds me of the fact that smartphone cameras nowadays use ML based filters to make faces prettier before saving in the first place.

Perhaps we can use clever experiment design to address it, like doing a "full-reference" evaluation (e.g., pick most similar two images, out of encoder A, encoder B, and original) rather than strictly A/B.

Its creator (me) mentioned the smoothing as a negative.