It is more surprising than that: it's because the humans in the loop like it.

If you're being given an A-B choice between cinnamon-flavored shit and lutfisk-flavored shit, you might convince the experimenter that cinnamon-flavored shit ranks highly in user preferences.

	▲	lukeschlather an hour ago \| parent [-]
		I would expect a preference for lutefisk flavored shit, the problem is then you then deduce from that that people prefer lutefisk flavor over cinnamon, rather than that you should stop feeding people shit (and also stop feeding them lutefisk.)

▲

twoodfin 6 hours ago | parent | prev [-]

The LLMs are going to turn us into stochastic parrots.