Yawn. Call me back when it's been reproduced.

Remember folks, we cannot cherry pick from the failed science of psychology even when it appears to be interesting.

By that metric all of science has failed, or at least the biomedical fields.

Psychology looks like particle physics compared to fields like dentistry, and not a lot different from a lot of biomedical fields.

Psychology is just the messenger — meta-science goes under the radar but it's probably its biggest contribution to science at large.

I agree about replication in any case (not just psychological studies), but I have to say this seems like it probably would. If anything I think it underrepresents the phenomenon it studies because support for corruption and unethical behavior is likely to be underestimated in this kind of setting/design.

	▲	kelseyfrog 3 hours ago \| parent [-]
		> By that metric all of science has failed, or at least the biomedical fields. How much should we weaken our epistemic standard in order to avoid difficult conclusions?

▲

7 hours ago | parent | prev | next [-]

[deleted]

▲

KittenInABox 7 hours ago | parent | prev [-]

I think we should be more generous, given the fact that we cannot reproduce many findings in our own field (computer science) given the hardware to do so is limited to such few entities...

▲

yjftsjthsd-h 7 hours ago | parent | next [-]

> given the fact that we cannot reproduce many findings in our own field (computer science)

Sounds like we should be less generous to CS findings, not more generous to other fields. (And yes, we should be skeptical of findings in CS as well.)

> given the hardware to do so is limited to such few entities...

That said - what research is happening in CS that needs specific hardware? The theoretical stuff can still happen on chalk boards, and interesting algorithmic or technical advances tend to propagate quickly precisely because someone will reproduce them.

▲

hearsathought 7 hours ago | parent | prev | next [-]

> I think we should be more generous

We really shouldn't.

> given the fact that we cannot reproduce many findings in our own field (computer science)

Computer science isn't a "science". Computer science is really a branch of mathematics. For example, when you study computation theory, you prove theorems (deduction). You don't generate a hypothesis and test it.

> given the hardware to do so is limited to such few entities...

Unless you are talking about computer engineering, which isn't really science either but engineering. Computer science isn't done in "hardware". Maybe you should go learn what computer science is.

▲

eirikbakke 7 hours ago | parent | next [-]

CS theory is indeed closer to mathematics. But other areas--database systems, computer architecture, networking, user interface design etc., is in fact evaluated via experiments, which is what makes it "science".

For example, if you propose some new technique to make databases faster (e.g. "store tuples column-wise instead of row-wise"), you'll implement it and run various workloads with and without the technique enabled. That gives you a quantitative measurement of the merit of the technique.

▲

bryanrasmussen 7 hours ago | parent [-]

I believe that falls under what the OP called engineering, which is fair as in engineering one often has to run experiments to determine if what one thinks will work in the particular case actually works.

	▲	eirikbakke 6 hours ago \| parent \| next [-]
		No, it's a whole scientific field, with experiments, lit review, publications, peer review and everything. The engineering artifact is just a by-product. More often than not, the code is thrown away or never used again, once the experiments have been run and the paper has been published.
	▲	dotnet00 5 hours ago \| parent \| prev [-]
		The distinction between engineering and science gets pretty blurred at the boundaries.

▲

js8 7 hours ago | parent | prev [-]

Not true, have seen many CS papers that run experiments lately. And arguably experimental mathematics is a science.

▲

lostmsu 7 hours ago | parent | prev [-]

Have you actually tried training a 10MB MoE (that would train in a few days on 3090)?

I came to an opinion that most of the current AI research can be easily reproduced on the small scale. CoT is possibly the only exception as it sounds like it requires certain emergent behavior, but even there I am not sure it is impossible to retrofit to tiny models.