It's not just the system prompt that's responsible; RLHF training based on user feedback can end up overly reinforcing "agreeable" behavior independently of the prompt. That's a big part of what got blamed for ChatGPT's sycophantic streak a few months ago.

> But there are other domains where it is catastrophically wrong, e.g. I had students come with an electronics proposal where the LLM misrepresented the relationship between cable gauge, resistance and heat in exactly the opposite way of what is true.

Since you mention that: I'm reminded of an instance where a Google search for "max amps 22 awg" yielded an AI answer box claiming "A 22 American Wire Gauge (AWG) copper wire can carry a maximum of 551 amps." (It was reading from a table listing the instantaneous fusing current.)