> The big problem is that a person making a mistake can be taught to not make that mistake again. That's also not foolproof but at least it works a lot of the times. AI are unteachable, if you have given them a good prompt and they do something wrong 90% of the time you are shit out of luck.

I feel like this line of thinking is kind of an unfair comparison. I'm not saying LLMs are magical beings that can suddenly learn by themselves after getting something wrong, but your "person making mistake then being corrected" assumes you do tell the person about the mistake and tell them to avoid doing the same mistake in the future, but for the "LLM making mistake" example you then intentionally avoid letting the prompt being changed in response to the mistake, which would be the "then being corrected" part on the LLM side of the comparison.

Similarly, if you just let a person make a mistake and don't let them know about the mistake, they might keep making that same mistake over and over again.

If you update how you use the LLM as you discover what mistakes it does, just like you'd correct a person, then you can use an LLM and also the LLM can "be taught to not make that mistake again".