I thought Claude got significantly smarter when I started using Rust. The big problem there is that I don’t understand the rust myself :P

It’s the style. Responses are always eloquent and well structured. When you look at output for a domain you don’t know well, you give it benefit of the doubt because it sounds like a highly competent human, so you react similarly. When you use it with something you know very deeply, you naturally look more for substance rather than form, and thus spot the mistakes much easier. This breaks most illusions of amazing reasoning abilities etc.

My ChatGPT is amazingly competent at gardening! Well, that’s how it feels anyway. Is it correct? I have no idea. It sounds right. Fortunately, it’s just a new hobby for me and the stakes are low. But generally I think it’s much better to be paranoid than gullible when it comes to confident sounding ramblings, whether it’s from an LLM or a marketing guru.