| ▲ | abdullahkhalids 6 hours ago |
| OpenAI has expressed this sentiment by not submitting to or saying they will submit their results to peer reviewed journals. |
|
| ▲ | fasterik 6 hours ago | parent | next [-] |
| I would say it's released in the spirit of open source. "Peer review" in the narrow sense exists primarily to assign prestige in academia; but there's nothing stopping anyone from "peer reviewing" the GitHub repository. |
| |
| ▲ | j2kun 6 hours ago | parent | next [-] | | I would say it's released in the spirit of machine learning's competitive landscape (which is the culture this emerged from). | | | |
| ▲ | a57721 5 hours ago | parent | prev | next [-] | | What kind of prestige? Peer review is anonymous unpaid work. A good review does not merely check the correctness of logical arguments, it gives suggestions for the exposition, citing the correct references, putting everything in the right context, etc. | | |
| ▲ | bananaflag 5 hours ago | parent | next [-] | | > What kind of prestige? Peer review is anonymous unpaid work. Prestige to the reviewed, not to the reviewer. | |
| ▲ | fasterik 5 hours ago | parent | prev [-] | | All of the reasons you listed for peer review are valid. The broader point is that peer review can happen outside of academic journals, and nobody has an incentive to submit to them who isn't trying to play the academic prestige game. For a significant example, see the history of Perelman's proof of the Poincaré conjecture. | | |
| ▲ | a57721 4 hours ago | parent [-] | | > journals are not the arbiter of truth and getting published in them is something only academics have an incentive to do Good, I just wanted to point out that peer review isn't primarily an arbitrage of truth, it is also to make sure the exposition is nice to read. When you get a reviewer who actually cares, you receive lots of feedback that isn't related to the correctness of Lemma 3.14.15 and stuff like that. |
|
| |
| ▲ | abdullahkhalids 5 hours ago | parent | prev | next [-] | | This is an equivalent of a company producing security software, open sourcing their code, and then claiming that since no one has found any serious bugs, their software is secure. No. The way to build confidence that your software is well made, you do a proper external security audit and obtain the requisite certificate from a proper auditing firm. It's also incorrect to think peer review in mathematics is low quality (like it is in some other fields). Certainly, when major results are in place, editors ensure that high quality peer reviewers are recruited and do their job properly. Like all human processes this fails sometimes, but not enough to not do it. | | |
| ▲ | john_strinlai 5 hours ago | parent | next [-] | | >then claiming that since no one has found any serious bugs, their software is secure. which specific openai statements does this part of your analogy map to? in the "sharing ai progress in mathematics" blog, openai simply says "results", and never once claims that all of them are unquestionably true. instead, they state they want to evaluate the results. their github states that the results are "different stages of verification" and also says "Some of the unformalized results could have issues" that is the opposite of "claiming [...] their software is secure", to use your analogy. | |
| ▲ | fasterik 5 hours ago | parent | prev [-] | | I didn't say peer review is low quality; just that it's not necessary or sufficient to determine the truth. Ultimately the OpenAI proof stands or falls on things that have been audited externally, namely the formalization of the problem in Lean and the correctness of the Lean software. There's no incentive for OpenAI to submit to a peer-reviewed journal when they don't need to play the academic prestige game. TFA is an example of peer review in action: they're analyzing the proof and finding points to criticize. |
| |
| ▲ | 1234-1298 5 hours ago | parent | prev [-] | | So they could also dump a 100 quadrillion line proof in Bourbaki notation and call it a day? The proof was released in the spirit of being first at all costs without any attempt to clean it up. I doubt that OpenAI mathematicians could give a coherent talk about it, certainly not using a blackboard. | | |
| ▲ | fasterik 5 hours ago | parent [-] | | Sure, why not? They can publish whatever they want, then the public can choose to ignore it, criticize it, or accept it. | | |
|
|
|
| ▲ | john_strinlai 6 hours ago | parent | prev | next [-] |
| not submitting to whatever journal is quite different than saying they are "beyond peer review" are people not reviewing openai claims right now? |
| |
| ▲ | j2kun 5 hours ago | parent [-] | | People described the problems as solved the minute they were made public. | | |
| ▲ | john_strinlai 5 hours ago | parent [-] | | this happens in approximately every scientific field. ive never heard it described as "idea that they are beyond peer review". openai themselves specifically call out that there may be issues with their results. journalists and laypeople just happen to skip that part, like they do with ~all physics, health, astronomy, etc results. |
|
|
|
| ▲ | TeMPOraL 5 hours ago | parent | prev | next [-] |
| You are confusing two levels of indirection here. Peer review is a proxy for correctness. Peer review journal is a proxy for quality peer review, or at least it was, once upon a time. |
|
| ▲ | yieldcrv 6 hours ago | parent | prev | next [-] |
| Because they want to release everything on github so everyone can peer review it themselves This is far more efficient and they’re telling the academic industry to grow up Sister comments are saying that academics dont like the Lean programming language and see a lack of human language described proof. Doesn’t sound like something I should care about but I’m watching for a better human language description of the problem as this discussion evolves |
|
| ▲ | setgree 6 hours ago | parent | prev [-] |
| "not interested in" != "beyond" |
