> If someone published a book advising people to take drugs, would people be filing lawsuits? No—we would agree that people are allowed to write whatever they want, even if what they say is terrible, right?

That's not the situation here. The more accurate case would be:

> If someone without a medical license provided blatantly incorrect medical advice with respect to safe medication usage to an individual via a direct one-on-one discussion, would people be filing lawsuits?

And the answer is yes. You can be wrong and you can say incorrect things. What you can't do is provide medical advice unless you are a licensed medical professional. You can still speak about medical topics but you have to disclaim your lack of licensure. You have to make it clear that you are not providing medical advice.

If this was a person doing this it'd be a crime, clear as day. It's called "practicing medicine without a license" and in the US it is a criminal offense in all 50 states, Washington DC, and all 5 inhabited territories. Whether it is a misdemeanor or a felony is dependent on the jurisdiction and the case but it's a crime everywhere in the US.

But ChatGPT doesn’t claim to have a medical license! You can give people whatever terrible medical advice you want—and people absolutely do—you just can’t claim to be a doctor!