I don’t know if it is valid. It is unverifiable. I still found some basic algebraic mistakes in top models as late as 3-4 months ago, not sure about it now. But that’s not what I want anyway, so I often put “Do not brute force” in my prompts.

▲

KoolKat23 2 hours ago | parent [-]

Sorry I mean in very public releases such as this trove, where many have lean certificates attached and publicly scrutiny.

▲

shakna an hour ago | parent [-]

Three of them have already been withdrawn. So I would say we have proof, that they cannot be implicitly trusted.

▲

KoolKat23 an hour ago | parent [-]

So it turns out there is a, perhaps informal, working system in place and we can deal with it.

Peer reviewed and published insights are proven invalid all the time. This is the nature of research and how we learn.

▲

shakna an hour ago | parent [-]

So instead of us "knowing it is valid", we don't. We need to put in extra effort, because someone felt like doing only half the work and dumping it on the community to fix.

We don't know the current system can work well enough at this scale, because that's un-knowable. We know it can find some of the problems. We don't know it can find all of them.

We do know it takes more effort - that's knowable. Increased data takes increased processing.

Whether the community has the required effort available, seems unlikely, considering the expertise required to be able to assess these things hasn't changed. Only the ability to generate them has increased.

	▲	KoolKat23 an hour ago \| parent [-]
		You don't have to go through it. You can ignore it if you wish. There is no obligation on you to check it. I feel your concern stems from the risk that there is additional noise everyone needs to cut through. In reality this isn't any tom, dick or Harry giving you their vibe code output. They have spent millions of dollars on this output, so there is a filter. The biggest filter of them all, funding. Furthermore, LLM's have given us another gift semantic search, we can easily check your work against theirs, this is valuable insight so instead of researchers wasting decades and fortunes pursuing an avenue that shows no value (this includes methods), they can purse new avenues they know what to avoid, in the same breath they know what to work towards.