Remix.run Logo
beepbooptheory a day ago

Because this is exactly what they controlled for. FTA:

> The researchers used Step 3.5 Flash, a model that was usually wrong on these questions, precisely so any reduction in judgment could not be explained as sensible delegation to a reliable tool.

(emphasis mine)

Dylan16807 6 hours ago | parent [-]

> precisely so any reduction in judgment could not be explained as sensible delegation to a reliable tool

That only works if they're experienced with model(s) of that level of unreliability and this is presented as one.

If they're used to a model that's more capable, and think the test model is similar, that's a huge confounding factor all by itself. It's not quite like giving fake credentials to a guy off the street and presenting them as an expert, but it's largely similar.