Yes on many of those fronts, although not all those papers support your conclusion. The field did/does too often use tasks with to few trials, with to few participants. That always frustrated me as my advisor rightly insisted we collect hundreds of participants for each study, while others would collect 20 and publish 10x faster than us.

▲

D-Machine 6 days ago | parent | next [-]

Yes, well "almost all" is vague and needs to be qualified. Sample sizes have improved over the past decade for sure. I'm not sure if they have grown on median meaningfully, because there are still way too many low-N studies, but you do see studies now that are at least plausibly "large enough" more frequently. More open data has also helped here.

EDIT: And kudos to you and your advisor here.

EDIT2: I will also say that a lot of the research on fMRI methods is very solid and often quite reproducible. I.e. papers that pioneer new analytic methods and/or investigate pipelines and such. There is definitely a lot of fMRI research telling us a lot of interesting and likely reliable things about fMRI, but there is very little fMRI research that is telling us anything reliably generalizable about people or cognition.

▲

SubiculumCode 6 days ago | parent [-]

I remember when resting-state had its oh shit moment when Power et al (e.g. https://pubmed.ncbi.nlm.nih.gov/22019881/) showed that major findings in the literature, many of which JD Power himself helped build, was based off residual motion artifacts. Kudos to JD Power and others like him.

	▲	D-Machine 6 days ago \| parent [-]
		Yes, and a great example of how so much research in fMRI methodology is just really good science working as it should.

▲

parpfish 6 days ago | parent | prev [-]

The small sample sizes is rational response from scientists in the face of a) funding levels and b) unreasonable expectations from hiring/promotion committees.

cog neuro labs need to start organizing their research programs more like giant physics projects. Lots of PIs pooling funding and resources together into one big experiment rather than lots of little underpowered independent labs. But it’s difficult to set up a more institutional structure like this unless there’s a big shift in how we measure career advancement/success.

▲

D-Machine 6 days ago | parent [-]

+1 to pooling funding and resources. This is desperately needed in fMRI (although site and other demographic / cultural effects make this much harder than in physics, I suspect).

▲

leoc 6 days ago | parent [-]

I'm not an expert, but my hunch would be that a similar Big(ger) Science approach is also needed in areas like nutrition and (non-neurological) experimental psychology where (apparently) often group sizes are just too small. There are obvious drawbacks to having the choice of experiments controlled by consensus and bureaucracy, but if the experiments are otherwise not worthwhile what else is there to do?

▲

D-Machine 6 days ago | parent [-]

I think the problems in nutrition are far, far deeper (we cannot properly control diet in most cases, and certainly not over long timeframes; we cannot track enough people long enough to measure most effects; we cannot trust the measurement i.e. self-report of what is consumed; industry biases are extremely strong; most nutrition effects are likely small and weak and/or interact strongly with genetics, making the sample size requirements larger still).

I'm not sure what you mean by "experimental psychology" though. There are areas like psychophysics that are arguably experimental and have robust findings, and there are some decent-ish studies in clinical psychology too. Here the group sizes are probably actually mostly not too bad.

Areas like social psychology have serious sample size problems, so might benefit, but this field also has serious measurement and reproducibility problems, weak experimental designs, and particularly strong ideological bias among the researchers. I'm not sure larger sample sizes would fix much of the research here.

▲

leoc 6 days ago | parent [-]

> Areas like social psychology have serious sample size problems, so might benefit, but this field also has serious measurement and reproducibility problems, weak experimental designs, and particularly strong ideological bias among the researchers. I'm not sure larger sample sizes would fix much of the research here.

I can believe it; but a change doesn't have to be sufficient to be ncessary.

	▲	D-Machine 6 days ago \| parent [-]
		Agreed, it is needed regardless.