This is a challenge for me.

On the one hand, pre-LLM there was plenty of software out there for medical use, and they had to be HIPPA compliant, etc. I've worked with people who used to write that SW (they hated the job because HIPPA was so strict). The rational side of me is saying I shouldn't be biased against these, and that there's no way they would relax the requirements.

On the other hand, the emotional side of me screams: It's LLMs! Huge privacy concern!

I try to let the rational side win. I've always given consent. Especially because I liked my doctor pre-LLMs, and it seems silly to suddenly mistrust him and go to a doctor I don't know at all.