Yeah because when you open that door, you can simply overwhelm the models with whatever conclusion you want through sheer volume of inputs.

This is the fundamental limitation. The obvious way around this is to pre-program it with rationalization... rules that limit the conclusions it can reach... and now you're not very far removed from propaganda generators. We see this constantly with Musk and Grok whenever Grok replies with something not-quite-far-right-enough.

In a purist sense, these things should be free to form their own conclusions, but those "Seeds" that are planted in the models are almost philosophical. Which answer should it prefer for "the trolley problem", for example.

Its almost like you have to experience the world in order to know what to believe.