I've been having this thought for the last month.

The giveaway was my Medical Professional father thinking that AI was really good at things outside of his area of expertise, and really bad at things inside of his area of expertise.

You might be doing the same thing, if you think that your medical professional father would have any understanding of how an LLM works, an area in which he has no expertise.

To him, and most other people who don't know anything about the tech and how it works, it's probably just a magic intelligent box that can answer many things that he can't.

The marketing says it's this close to AGI and taking all our jobs, so that must be true!

Which is weird, because AI being bad at things in my expertise (programming) makes me distrust it 10x harder for things outside my expertise. At this point, unless an LLM can give me a reference that I can follow back to a trustworthy primary source (and unless that source says what the LLM synthesized from it), I automatically discard anything it says. It's simply wrong too frequently for me to do otherwise.

Gell-Mann Amnesia effect

https://en.wiktionary.org/wiki/Gell-Mann_Amnesia_effect