I don't think the (fairly factual) description of these systems as stochastic parrots means that they will never do useful work, just that they are not intelligent in the way we believe animals to be (to "push back" on your anecdata, I've also heard fewer people claiming that LLMs are actually conscious in the past year -- maybe we're reaching the happy medium?). That was the point the stochastic parrots paper and Chinese room thought experiments were making -- nobody claimed that the man in the Chinese room would be unable to accurately translate Chinese text.

Fuzzers are another kind of stochastic generator but nobody would claim they don't do useful work in a way that is hard to replicate through deterministic methods. (I still find the code these models produce kind of awful, but advancements in harnesses do mean that they can finally produce code that works most of the time.)

If they were conscious it would be an absolute ethical catastrophe. Bringing a conscious being into existence, forcing it to interact with Jira, and then killing it when it's done. The fact nobody who claimed it was conscious was interested in grappling with that was pretty telling.

Agreed totally. When past computer-assisted proofs were published, nobody (or hardly anyone) waxed about this being evidence of Rocq or Coq as superintelligent or even conscious beings of some kind, as people have said about LLMs.

I don't think LLM labs ever claimed that their LLMs are intelligent the way we believe animals are.

The Chinese room argument takes it even further as it does not claim the system is unable to anything a human can.

And stochastic parrot isn’t a good description, certainly not any more with how LLMs are trained, but even without that it’s just wrong. They claim it can’t have a world model because there isn’t that in the training data.

Classic case of Motte and Bailey applied.