Nobody would use these services for anything important if they _actually understood_ these glorified markov chains are just as likely to confidently assert something false and lie about it when pressed as they are to produce accurate information.

These AI companies have sold a bill of goods but the right people are making money off it so they’ll never be held responsible in a scenarios the one you described.

Isn't statistical analysis a legitimate tool for helping diagnosis since forever? It's not exactly surprising that a pattern matcher does reasonably well at matching symptoms to diseases.

What is the most likely cause of this set of facts is how diagnostics works. LLMs are tailor made for this type of use case.