In fields like AI/ML, I'll take a preprint with working code over peer-reviewed work without any code, always, even when the preprint isn't well edited.

Everyone everywhere can review a preprint and its published code, instead of a tiny number of hand-chosen reviewers who are often overworked, underpaid, and on tight schedules.

If the authors' claims hold up, the work will gain recognition. If the claims don't hold up, the work will eventually be ignored. Credentials are basically irrelevant.

Think of it as open-source, distributed, global review. It may be messy and ad-hoc, since no one is in charge, but it works much better than traditional peer review!

▲

smokel 4 days ago | parent [-]

I sympathize partially with your views, but how would this work in practice? Where would the review comments be stored? Is one supposed to browse Hacker News to check the validity of a paper?

If a professional reviewer spots a serious problem, the paper will not make it to a conference or journal, saving us a lot of trouble.

▲

yorwba 4 days ago | parent | next [-]

Peer review is a way to distribute the work of identifying which papers are potentially worth reading. If you're starting from an individual paper and then ask yourself whether it was peer reviewed or not, you're doing it wrong. If you really need to know, read it yourself and accept that you might just be wasting your time.

If you want to mostly read papers that have already been reviewed, start with people or organizations you trust to review papers in an area you're interested in and read what they recommend. That could be on a personal blog or through publishing a traditional journal, the difference doesn't matter much.

▲

conception 3 days ago | parent [-]

“Find papers that support what you want via online echo chambers” isn’t the advice you want to be giving but it is the net result of it. Society needs trusted institutions. Not that publishers are the best result of that but adhoc blog posts are decidedly not better.

▲

yorwba 3 days ago | parent [-]

It is totally the advice I want to be giving. Given the choice between an echo chamber matched to my interests and wading through a stream of unfiltered crap, I'll take the echo chamber every time. (Of course there's also the option of not reading papers at all, which is typically a good choice if you're not a subject matter expert and don't intend to put in the work to become one.)

If you choose to focus on the output of a well-known publisher, you're not avoiding echo chambers, you're using a heuristic to hopefully identify a good one.

	▲	ricardobeat 3 days ago \| parent [-]
		Those are not the only options, namely the parent mentioned 'trusted institutions'. It is the best way to defer that filtering to a group of other humans, whose collective expertise will surpass any one individual. The destruction of trust in both public and private institutions - newspapers, journals, research institutions, universities - and replacement with social media 'influencers' and online echo chambers is how we arrived at the current chaotic state of politics worldwide, the rise of extremist groups, cults, a resurgence of nationalism, religious fanaticism... This is terrible advice.

▲

belter 4 days ago | parent | prev | next [-]

> If a professional reviewer spots a serious problem

Did that ever happen? :-)

	▲	atq2119 4 days ago \| parent [-]
		Of course. As usual, you tend to not hear about it when a system we rely on works well.

▲

rstuart4133 2 days ago | parent | prev [-]

Your question is like asking "how can I verify this rod is 1m long if I can't ask an expert". The answer is of course you measure it. That's much more reliable than asking an expert. However, the results of many papers take a huge amount of work to replicate, so we've built a network of experts over the years to evaluate them.

But this is open source, so TL;DR: you download the code, run it, and see if it gets the results claimed.

	▲	smokel a day ago \| parent [-]
		Have you tried that with the code repository that we're discussing here? It took this trained professional over an hour to get started, and then I gave up. It would take an additional 24 hours and quite some hardware simply to reproduce the results, and then probably a few weeks to actually understand what is going on. All in all, not very practical.