There exist many problems for which it's easy to verify the inputs-outputs, but much harder to write the functions to convert the inputs into the outputs.

Just write an executable spec and have the AI generate the code that fulfills it. Where is the risk?

I guess the risk would be that the cost of compute would explode out of control long before you got a viable solution.

To give a bit of an absurd example, I imagine there's a reason it's not currently worth it to generate the code with a random character generator, even though hypothetically that would get you there eventually. If we consider AI a much, MUCH better version of a random character generator (let's say it's a million times faster. No, let's say that it'll get you the solution in quadratic time instead of factorial time), that doesn't necessarily mean it's actually worth it now.