It'd be interesting to scale this up to a lot of images - and have users blind A/B test, like the LLM Arena on Huggingface which helped filter out "benchmaxxing" vs actual intelligence & utility. Because 1) images can be so diverse* and 2) an intrinsic tradeoff might exist between general efficiency and out-of-distribution robustness.

It's hard to rigorously judge a codec from a handful images, especially given the large impact they would have (with a widely adopted codec encoding bazillions of images in the coming years).

* There are many intricate details that go into codecs and visual perception. For instance, our sensitivity to visual errors are dependent on colors (all of hue, saturation, and brightness), spatial frequency of artifacts, local contrast and texture presence, et cetera.

And like OP mentioned, perceptual error metrics (including SSIMULACRA2) aren't perfectly representative of human-evaluated visual quality. For instance, libjxl deliberately makes choices that hurt SSIMU2 scores, but improve human-rated quality.

▲

janwas 3 hours ago | parent | next [-]

There was a large scale study a while back: https://cloudinary.com/blog/contemplating-codec-comparisons

	▲	ksec 2 hours ago \| parent [-]
		That was done before AVIF had any improvements. Arguably one could even say JPEG XL competition brought the best of AVIF out.

▲

jaffathecake 5 hours ago | parent | prev [-]

There was a study done while I was at Google, and I fought for it to be done on high density screens, because that's what most users have. The response was "but people won't be able to see the artifacts" and my head nearly rotated 180 degrees.

But yes, you can see this with jixel, which I'm really impressed by, but it's so tuned to SSIMU2 that it looks really bad to human eyes in some cases compared to libjxl.

▲

eviks 4 hours ago | parent | next [-]

> but people won't be able to see the artifacts

But that is correct and why any proper study should first train the participants on the type of artifacts and trade-offs. Like the reference to JXL choices: think its creator mentioned that AVIF might look nicer for faces because it was "smoothing" them but it was a worse representation of the original. So the study participants would need to be conscious of that trade off and be clear in terms of what they're asked to evaluate - whether the result looks nicer or whether it represents the original better.

	▲	yhjc2692 4 hours ago \| parent \| next [-]
		This is a good food for thought. Semi-related: It reminds me of the fact that smartphone cameras nowadays use ML based filters to make faces prettier before saving in the first place. Perhaps we can use clever experiment design to address it, like doing a "full-reference" evaluation (e.g., pick most similar two images, out of encoder A, encoder B, and original) rather than strictly A/B.
	▲	jaffathecake 4 hours ago \| parent \| prev [-]
		Its creator (me) mentioned the smoothing as a negative.

▲

yhjc2692 4 hours ago | parent | prev [-]

I feel that!

Maybe the first step to designing better encoders is to build an extremely accurate visuoperceptual error metric. Seems like ML's rapid progress haven't caught onto it just yet.

▲

janwas 3 hours ago | parent [-]

This was indeed what the JPEG XL effort started with :) The Butteraugli metric.

	▲	yhjc2692 2 hours ago \| parent [-]
		Haha, that's a great point! On that note - do you have any good pointers on understanding more about how Butteraugli works? I wholeheartedly agree it is a great metric (which aligns well with my subjective assessment much better than MSE or MS-SSIM), but the Guetzli paper (or the Butteraugli GitHub repo) does not explain the theory behind it in detail. Seems like others were curious too, given the Wikipedia page on Guetzli mentions: > Butteraugli. ... How the hundreds of parameters that model the properties of the human visual system were derived remains unexplained. My rough understanding is its multi-scale convolution architecture was inspired by biological visual perception (perhaps how the layers of visual cortex gradually represent higher level features, ish), and involved some sort of empirical parameter tuning based on human evaluations. It'd also be a very nice resource for beginners wanting to learn more about image/video codecs (like me :)