Sure, I agree with point (3).

But for (1, 2), just to point out, neither of your criteria would pass muster in the scientific communities who are ultimately responsible for vetting this work, right?

The ideal is: we don't accept papers at NeurIPS (or wherever) because of affiliation, and the badge of "quality" is bestowed by peer review, from other scientists.

> is of comparable quality (if not better than)

But the posts that you refer to in your own post point out several ways in which the quality is not good ...

But, in any case, I find it a bizarre choice to call it a paper when the authors themselves do not refer to it as such. We can agree to disagree, but just offering feedback.

▲

simonw 2 hours ago | parent [-]

You're right about this, but I don't regret my choice of words.

I admit that I don't think formal "papers" deserve their elevated status in today's information economy.

I think they add a whole lot of ceremony that usually results in a format that's harder to understand.

So me calling this a "paper" was a little bit of a dig!

▲

mccoyb 2 hours ago | parent [-]

Of course, I agree with you in many respects - but I’ve found the academic paper lifecycle to be tremendously helpful in terms of polishing my contributions (admittedly, not in ML - but in PL / systems) and avoiding content which could be construed as “misleading”

You, being of sound mind, tried the technique and drew reasonable conclusions … but just look at the original post for this blog post: there are a large number of responses by people who clearly don’t understand the technique, missed the experimental weaknesses, couldn’t place the post into the lineage of mech interp results that preceded it … and will leave the discussion with the wrong impression.

In your mind, whose fault is that? Is that that the unavoidable cost of a field, moving too rapidly to engage in peer review?

In its ideal form, the paper lifecycle is not a gate, but a sharpening stone.

▲

simonw 2 hours ago | parent [-]

I think we have a messy version of peer review playing out right now.

I'm not convinced the alternative - delaying publication by 6 months, keeping the new information available only to a much smaller group until then - would have been better.

All of that said, I did just update my own post to include "The paper (OK, the authors classify is as a “blog post”, but it’s paper-shaped) found that frontier models..." inspired by this conversation.

	▲	mccoyb 2 hours ago \| parent [-]
		Agreed! Now if the authors update their post in response — we’ll have a proper venue on our hands!