There’s more than two options here. It was already difficult to deal with self diagnosis for doctors, now we have a machine that outputs recommendations, and does it with confidence whether it’s correct or not.

The same issues that were present with search-engine self diagnosis are still present with LLMs. If you provide Google with an incomplete list of symptoms and can’t interpret the information you find correctly, you will likely get an incorrect diagnosis. The same is true for LLM output.

There are quite a few disclaimers everywhere that soften confidence: "always ask a medical specialist", "I'm not a doctor", "this could have been this or that but really not sure", etc.