I think it's more like lottery winners giving "buy lottery tickets" as financial advice.

It is clear at this point that any meaning found in LLM outputs is projected there by the user. Some people by virtue of several intertwined factors can get some acceleration out of them, but most can't. It becomes like a football fan convinced that their rituals are essential for the team's victory.

Add the general very low understanding of machine learning (or even basic formal logic) and people go from a realistic emulation of conversations to magical thinking about having a mind in a box.

Either that or I am taking crazy pills. Because sometimes it feels like that.