It feels like moving toward a local maxima. In this example the LLM advice outperformed one doctor who dropped the ball in some ways, but I do not look forward to a world where we make it even more difficult to talk to doctors, and route most medical interactions to LLMs. I expect that in 5 years I'll be futilely shouting "representative" into my phone as my appendix ruptures.

With this sort of advice from LLMs, I think there's a lot of selective recall that's easily overlooked -- where we gloss over all the bad, dead-end lines of advice we seek from the LLM oracle.