none of the large AI labs know what the best outcome is because lab science never fully predict large scale implementation.

We've had decades of grade A research on social improvement programs and it's clear to everyone that implementation cannot be simulated or easily predicted.

So the "optimizing" thing is a fallacy. It's also found in highly racially charged pseudo science and eugenics; selecting for trains on small scales do not equal large scale benefits. But people's mental model are predisposed to small scales while also being blissfully ignorant of cognitive biases to the predisposition.

Even the idea that we can just keep throwing compute, context, power is a fallacy when you see chinese models making smaller and just as capable models with limited resources.

so, optimizing is something we should be putting into a democratic process because anything else will be lopsided, much like when eugenics was tried.