LLMs outputting code that doesn't compile is the failure mode you want. Outputting wrong code that compiles is far worse.

Setting aside the problems of wrong but compiling code. Wrong and non-compiling code is also much easier to deal with. For training an LLM, you have an objective fitness function to detect compilation errors.

For using an LLM, you can embed the LLM itself in a larger system that checks it's output and either re-rolls on errors, or invokes something to fix the errors.