I paid a bit of attention to this paper and the phrase 'stochastic parrots' when it came out and i thought this was worth saying and doing at that time. their suggestions about financial and environmental costs are worth studying, their concern about carefully evaluating datasets to feed to the model rather than feeding the entire internet is fully justified. so - to everyone saying this was a bad paper; if you have actually read the paper then please list a few criticisms. all i have seen is "oh this wasn't that good of a paper" or "can't believe how bad this paper was".

▲

azakai an hour ago | parent | next [-]

My main criticism of the paper is that it says LLMs work "haphazardly", using probabilistic information. That is a hypothesis, but it is stated as a known fact, a fundamental limitation.

It is true that LLMs often behave haphazardly, and do rely on statistics. But plenty of research has shown them behaving in methodical ways too. There are findings going both ways!

Granted, many of the strongest contradictory results appeared after the Stochastic Parrots paper, so it isn't like they were ignoring the literature at the time. But they did make a very strong claim, and in the half-decade since, a lot of evidence has come out against it.

▲

dwa3592 an hour ago | parent [-]

not sure your criticism makes sense though - they did this pre chatgpt. they are talking about the language models of that time. they did not make predictions about the future.

	▲	azakai 10 minutes ago \| parent \| next [-]
		They made a claim about language models in general, not just ones that had been released so far. The point of the paper, in fact, is that language models are getting "too big", and another approach is needed to make progress, so they were certainly predicting things about later models. With that said, they talked about "pure" language models, so it is fair to say that they didn't talk about, say, LLMs that are multimodal or that have tool use, which are advances that happened after their paper.
	▲	mediaman an hour ago \| parent \| prev [-]
		They did use RLHF at the time, at which point it is not a pure probabilistic representation of the training corpora. Bizarrely, RLHF never came up in the paper.

▲

CamperBob2 3 hours ago | parent | prev | next [-]

Those costs have to be compared to the way things are currently done without AI.

They never are. Ever.

▲

matusp an hour ago | parent | prev [-]

It is a good blog, not a good paper.