Part of the problem to me is that these models are so damned agreeable. I haven't used ChatGPT in a while, but Claude is always assuming I'm right whenever I question something. I have to explicitly tell it not to assume I'm right, and to weigh my question with what it suggested. Maybe if they were trained to treat questions more skeptically, this kind of thing wouldn't happen.

And they're so "friendly"! Maybe if they weren't so friendly, and replied a little more clinically to things, people wouldn't feel so comfortable using them as a poor substitute for a therapist.

I really want the LLMs to respond like a senior developer that doesn't have time for you but needs you to get your job done right. A little rude and judgemental, but also highly concise.