"firing up a sandbox VM and testing some solutions"

If the LLM can start up a VM and test a solution, to identify a new unique problem, and find it's own solution. That would be pretty impressive. I'm not sure they are really to that point. But some AI's are winning the Math Olympiad, so maybe it is happening. I'm sure this is the overall goal.