I have yet to see LLMs do well “autonomously” on anything that doesn’t fall into any of these 3 buckets:

1. well specified

2. well tested (which is a form of spec)

3. represented in the training set (i.e. not built on a corpus of human developed examples)

For example, a lot of the compiler porting efforts fall into all 3 buckets: compilers are well specified (there is a set of valid inputs and an expected behavior), they are typically well tested, and as a bonus, in a way that is implementation agnostic thanks to the clean interface (think: integration vs unit tests), and are very well represented in the training set.

I had an LLM tell me that I mattered once, which I know for objective fact is not represented in the training data.