Discovery of the best solution in a problem space is not generative but only verificative. Meaning: the LLM can see if a solution is better than another, but it can't generate the best one from the start. If you trust it, you'll get sub-par solutions.

This is definitely an agent problem instead of an LLM problem. Anybody got something explorative like this working?

So? Hundreds of millions of office and devel jobs are about for developing "optimal solutions" to begin with.