IMHO the only defensible way to benchmark this is to use secret-shoppers via subscriptions. As is standard practice in other product-review fields, where the stakes are a lot lower.

The purpose of AI benchmarking is to inform people making large spending decisions what alternatives are available to them.

I *do* wish we wouldn't let software companies call this "A/B testing". That's not what that term means to a typical reader. In any other market than software, you'd call it something like "misleading consumers in respect to product composition" and "an unfair commercial practice". That's language from the EU "dual quality" food scandal, which saw food brands (Coca-Cola was one) adulterate their recipes to use cheaper ingredients in select markets—though the external packaging was kept the same.

https://ec.europa.eu/commission/presscorner/detail/en/qanda_...

(I suppose Coca-Cola's CEO would prefer we talk about what they did as "A/B testing").