This is exactly my point. They are not repeating experiments. Each one is a different one, and independent of the other, because the chemicals in each color are different. The cartoon don't show any data that proves that this is a random correlation, because they are not testing what you think that they are testing. Jellybeans here are irrelevant (They are present in both groups, the control and the experiment). It may be, or not be, a random correlation, we can't know at this point. And this leads us directly to the next line:

> A positive result is no more deserving of further investigation than a negative result.

This is not how science works.

Your null hypothesis is rejected, you publish and continue trying to explain why. Otherwise you stop and try something different.

Films have sold us the idea that science can be done in the countdown of a bomb, experiments take 30 seconds; and if it does not work in the first try, it means that you are not a real genius. The weird guys build a time machine in a weekend with empty cans of soda, and the hackers broke into high security buildings with a paperclip and then remove their glasses and nobody recognizes then anymore. Yeah. Master of disguise. This Big Bird costume pays for itself every time we break into the Pentagon. G'morning Mr Hegseth

But this is a fantasy, not the real life. Scientists stumble and limp all the way. Even the geniuses. Specially the geniuses. And they faint at the price of the required spare part that must be ordered and imported from one, and only one, company in Finland, and will be here in five months. Meanwhile, researchers throw things to the wall to see what sticks, and laser focus on the unexpected strange result that is different in a subtle way, because their grants depend on it.

▲

dpark 3 days ago | parent [-]

> This is exactly my point. They are not repeating experiments. Each one is a different one, and independent of the other, because the chemicals in each color are different. They are not testing what you think that they are testing.

They ran the same experiment 20 times with a meaningless variable changed until they got a positive result with a confidence of 95%. (i.e. 1/20 chance of a random positive result.)

The point of the comic is that this is bad science. Or at least it is ver bad science reporting. This kind of nonsense is why there is a multibillion dollar market for supplements that do not hold up to further testing.

> This is not how science works in the real life.

True. There is a lot of bad science in real life.

▲

pvaldes a day ago | parent [-]

"Scientists bad because I don't understand statistics" is boring. Just because you decided that is a meaningless variable does not mean that it is.

P-hacking doesn't mean doing a lot of experiments. This is called working hard or going the extra mile. Working hard is not cheating. That the real scientists would do is frown upon the green jellybean result, think "hum, this is strange" and test further for effects of the green dye in other products. Science knowledge is in-cre-men-tal.

P-hacking is: changing the statistical model that you use to analyze the data, until you find one that gives you the desired (publishable) p-value.

P-hacking is manipulating the methodology in your favor and sometimes grooming the data also. All of that while ignoring other models available known to be more robust, precise and better for your particular case. This is why p-hacking is cheating.

-------------------------

Second. There are a market for fake supplements because, to express it crudely: people is idiot and the government does not care about them.

1) Some people didn't cared to learn even basic biology at the school, and will believe that the moon is made of cheese if it comes from somebody in their tribal team. Scientists take a lot of pain to assure that their knowledge is available to the public.

There are thousands of scientific articles about creatine. Never has been so easy in the history to find them. And with AI, you don't even need to read the whole horror anymore, to have a general idea of cons and pros before to jump in the water. Do your f*k homework before to gulp a mysterious pill. If not. Don't blame scientists later.

2) The government make laws. Is not scientists fault, if it allows to sell legally dubious supplements, or even more if politicians profit of it, (ehum, mandated testosterone tests for every soldier, ka-ching!).

Their job is to protect citizens from scammers. Most countries ban anything that is remotely medical and not proven effective, or at least advice strongly against taking it.

	▲	dpark a day ago \| parent [-]
		> "Scientists bad because I don't understand statistics" is boring. Just because you decided that is a meaningless variable does not mean that it is. It’s literally a comic about how spurious results arise. I understand what you are trying to get at with your insistence on pedantically misreading the comic, but you’re missing the forest for the trees. > Your null hypothesis is rejected, you publish and continue trying to explain why. In this case the “why” is “we ran so many experiments that a false positive was inevitable”. > Otherwise you stop and try something different. “Only publish the positive results” is exactly the problem. This is terrible science because it magnifies false positives and hides the negatives (false or not). Do enough experiments and eventually you get a positive result for basically anything.