While they absolutely do fail as you say (though in my experience not by default), this failure mode is still a massive improvement over the frequent human case of guessing based on the function/class/property/argument names.

Now, a really good human collaborator who reads all the stuff and thinks carefully, that was still better than what I saw from AI models at the start of this year. But I've also worked with my share of idiots, and been one too.

I'm not going to get into if *current* models can or can't reliably do any particular thing to any particular standard; previously my comparison was the same conversations with regard to video game computer graphics in the 90s always being "photorealistic" when they really weren't*; now, I'm starting to feel such discussions have the same vibes as Tesla fans insisting that "FSD-{insert current version here} solves all the problems and is a real breakthrough and the Rototaxi will totes conquer the marketplace this time for real bro, just one more version bro", etc.

* https://archive.org/details/nextgen-issue-26