LLMs are pretty much the exact opposite of a reliable compiler-like abstraction.

This is true to such an extent that I have to question the overall competence of anybody who makes the comparison. It's an enormous red flag.

I'd recommend reading Joel spolsky's leaky abstractions essay coz while it applies less and less 20 years later to things like kernel abstractions it explains very well why treating the LLM as a compiler sets you up for abject failures.