I’ve found Claude (at least until Opus 4) would routinely fail at writing a bash script. For example it would end an if block with }. Or get completely lost with environment variables and subshells.

But those are exactly the same mistakes most humans make when writing bash scripts, which makes them inherently flaky.

Ask it to write code in a language with types, a “logical” syntax where there are no tricky gotchas, with strict types, and a compiler which enforces those rules, and while LLMs struggle to begin with, they eventually produce code which is nearly clean and bug free. Works much better if there is an existing codebase where they can observe and learn from existing patterns.

On the other hand asking them to write JavaScript and Python, sure they fly, but they confidently implement code full of hidden bugs.

The whole “amount of training data” is completely overblown. I’ve seen code do well even with my own made up DSL. If the rules are logical and you explain the rules to it and show it existing patterns, the can mostly do alright. Conversely there is so much bad JavaScript and Python code in their training data that I struggle to get them to produce code in my style in these languages.