| |
| ▲ | 9rx 4 days ago | parent | next [-] | | 1. Papers are written to invite replication — the most important part of the scientific process. It is already difficult to compel replication even when you only put the most promising research in people's faces. Now you want them to have to also sift through thousands of entrants that have almost no reason for replication attempts? 2. It's easy to call it shameful when it isn't you who has to do the work. If you are like most other normally functioning people, you no doubt perform little experiments every day that end up going nowhere. How many have you written papers for? I can confidently say zero. I've got better things to do. | | |
| ▲ | jraph 4 days ago | parent | next [-] | | The lack of null result publication is a real issue for research and scientific knowledge in general, including for accurate positive results (what if once you see a positive result by statistical chance but the experiment was actually conducted before you by 10 other people but that wasn't published because the results were null?) > Papers are written to invite replication They are written to share knowledge, new discoveries. We hope that they are replicated. > It's easy to call it shameful when it isn't you who has to do the work. We are not judging anyone, we are qualifying the situation. And we are speaking of publishing results for experiments that have already conducted, not voluntarily making up null cases and doing the related experiments. It wouldn't make sense and would be harmful. But if you did an experiment that produced a null result, not publishing is a loss of knowledge. Again, we are not judging anybody, it's a failure of the publishing system. But this cannot change if nobody points it out. I'd have troubles understanding a researcher not acknowledging today that the lack of null result publication is an issue. It would show a lack of perspective IMHO. And for someone acknowledging that there's an issue, pushing back is not the right stance. | | |
| ▲ | 9rx 4 days ago | parent [-] | | > They are written to share knowledge, new discoveries. We hope that they are replicated. If they aren't replicated, no new knowledge is gained. Not really. > it's a failure of the publishing system. What failure? Again, the "publishing system" is to submit the "best of the best" research to the world in order to invite replication. While nothing is perfect, we don't want people to have to sift through papers that have effectively no reason to be replicated in a quest to find something that is. That would make things infinitely worse. The internet was created for the minor-leagues. If you are willing to put in the effect to document your "less than the best" research, put it up on a website for all to see already. There is nothing stopping you. But who wants to put in the effort? > We are not judging anyone How do you define "shameful" if it isn't related to judgement of someone? | | |
| ▲ | jraph 2 days ago | parent [-] | | You assume that null results are not worth it and are not the best of science. You seem to be ignoring the issue where researchers collectively waste time redoing "failing" studies again and again because the null results are not published, and other issues not publishing them causes. We fundamentally disagree here and I'm not willing to put in the effort needed to try to convince you otherwise, many other comments are here are better than what I could write. I do hope you take the time to take a step back (and you seem to be in defensive mode, if so, you need to go out of this) and reconsider. > How do you define "shameful" if it isn't related to judgement of someone? Do you know the expression "it's a shame"? > used when you wish a situation was different, and you feel sad or disappointed https://www.ldoceonline.com/dictionary/it-s-a-shame-what-a-s... | | |
| ▲ | 9rx 2 days ago | parent [-] | | > You assume that null results are not worth it and are not the best of science. No. I make no such assumption. That may often be true, but there is nothing to stop a useful null result from being published. We also get things wrong from time to time. It is very possible that the best baseball player in the world has been overlooked by Major League Baseball. We're pretty good at scoping out the best, but nothing in life is perfect. If the best baseball player in the world ends up in the minor leagues instead, oh well? Does it really matter? You can still watch them there. Same goes for research. If something great doesn't make it into the formal publication system, you can still read it on the studier's website (if they put in the effort to publish it). > You seem to be ignoring the issue where researchers collectively waste time redoing "failing" studies again and again because the null results are not published I may be ignoring it, but if that's the case that's because it is irrelevant. There is no reason to not publish your "failing" studies. That's literally why the public internet was created (the original private internet was created for military, but next in line was university adoption — to be used for exactly that purpose!). > We fundamentally disagree here and I'm not willing to put in the effort needed to try to convince you otherwise Makes sense. There is nothing to convince me of. I was never not convinced. But it remains: Who wants to put in the effort? Unless you are going to start putting guns to people's backs, what are you expecting? | | |
| ▲ | jraph a day ago | parent [-] | | > I was never not convinced Ok. > There is no reason to not publish your "failing" studies. That's literally why the public internet was created You are suggesting researchers should blog about their null results? It seems to me the null results deserve the same route as any other paper, with peer reviews, etc. It matters, because this route is what other researchers trust. They wouldn't base their work on some non reviewed blog article that can barely be cited. You don't even base good science on some random article on Arxiv that was not published in some recognized avenue. If you are using some existing work to skip an experiment because it tells you "we've already tried this, it didn't show any effect", you want to be able to trust it like any other work. Hell, as a random citizen in a random discussion, especially one with a PhD, I don't want to be citing a blog article as established scientific knowledge. And yes, getting published in a proper, peer reviewed avenue is work, but we all need to deeply internalize that it's not lesser work if the result is null. > Unless you are going to start putting guns to people's backs, what are you expecting? If researchers collectively decide it's worth pursuing, it's all about creating the incentives at the right place. Like any other research, you could be rewarded, recognized and all. High impact journals and conferences could encourage researchers to publish / present their null results. Of course, we are not speaking about such things like "what two unrelated things could I try to measure to find some absence of correlation", we are speaking about "I think those two things are linked, let's make an experiment. Oh, no, they are not correlated in the end!" -> the experiment is done either way, just that the results also deserve to be published either way. And the experiment should only be published if it doesn't exhibit a fatal flaw or something, we are not talking about flawed experiment either. | | |
| ▲ | 9rx a day ago | parent [-] | | > You are suggesting researchers should blog about their null results? If they want to. Especially if it doesn't meet the standard for the publication system, why not? > It seems to me the null results deserve the same route as any other paper, with peer reviews, etc. If it ranks with the best of them, it is deserving. There isn't room for everything, though, just as there isn't room for everyone who has ever played baseball to join the MLB. That would defeat the entire purpose of what these venues offer. But that doesn't mean you can't play. Anyone who wants to play baseball can do so, just as anyone who wants to publish research can do so. > If researchers collectively decide it's worth pursuing It only takes an individual. Unlike baseball, you can actually play publishing research all by yourself! 1. Where do we read your failed research? Given your stance, it would look very foolish to find out that you haven't published it. 2. Do you draw a line? Like, if you add a pinch more salt to your dinner and found that it doesn't taste any better, do you publish that research? | | |
| ▲ | jraph a day ago | parent [-] | | > There isn't room for everything I get your point, but this is not specific to null results. > It only takes an individual No no no. The desirability of null results need to be recognized and somewhat consensual, and high impact journals and conferences needs to accept them. Otherwise, there's no reason researchers will work to publish them. 1. I don't publish anymore: I'm not a researcher anymore. I didn't encounter the case during the short time I was one (I could have, though. Now I know, years later. I suspect it would have been difficult to convince my advisors to do it). I hope this doesn't matter for my points to stand on their own. Note that I think null results ARE NOT failed research. This is key. 2. Ideally, null or positive result alike, the experiments and the studies need to be solid and convincing enough. Like, there needs to be enough salt and not too much, the dinner needs to be tasty in both cases. If the dinner doesn't taste good, of course you don't publish it. There is something wrong with what you've done (the protocol was not well followed, there's statistical bias, not enough data points, I don't know) It feels like we are talking past each others, you are thinking I'm talking about failed research, but I'm talking about a hypothesis you believed could be true, you built an experiment to test it, and found no correlation in the end. This result is interesting and should be published, it's not failed research. As it happens, I attended a PhD defense less than a month ago where the thesis lead to null results… The student was able to publish, these null results felt somewhat surprising and counter intuitive, so it's not like it's impossible, it just needs to be widely seen as not failed research. | | |
| ▲ | 9rx a day ago | parent [-] | | > The desirability of null results need to be recognized and somewhat consensual If it is interesting you should also find it interesting when you read it 30 years in the future. You don't need other people. It's a nice feeling when other people want to look at what you are doing, sure, but don't put the cart before the horse here. Publish first and prove to others that there is something of value there. They are not going to magically see the value beforehand. That is not how the human typically functions. It's not like you have to invent the printing press to do it. Putting your work up on a website for the entire world to see is easy peasy. Just do it! > Ideally, null or positive result alike, the experiments and the studies need to be solid and convincing enough. No need to let perfect become the enemy of good. Publishing your haphazard salting experiment isn't apt to be terribly convincing, but it gets you into the habit of publishing. Eventually you'll come around to something that actually is interesting and convincing. It's telling if someone isn't willing to do this. > The student was able to publish, these null results felt somewhat surprising and counter intuitive, so it's not like it's impossible Exactly. Anything worthy of the major leagues will have no trouble getting formally published. But not everything is. And that's okay. You can still publish it yourself. If you want to play baseball, there is no need to wait around for the MLB to call, so to speak... Just do it! > you are thinking I'm talking about failed research [...] it just needs to be widely seen as not failed research. Yes, I am talking about what is widely seen as failed research. It may not actually be failed research in a practical sense, but the moniker is still apt, especially given that you even call it that yourself. I guess I don't understand what you are trying to say here. | | |
| ▲ | jraph a day ago | parent [-] | | > I guess I don't understand what you are trying to say here I guess I failed to get my point across to you and I doubt I will suddenly manage to do it this far in the discussion. Nature failed this too apparently, so I guess that doesn't tell much about me. > Putting your work up on a website for the entire world to see is easy peasy. Just do it! I have already said why this is not an option. I'm terribly confused as to why you are even suggesting it. Recognized research doesn't currently happen in blog posts. > it gets you into the habit of publishing. Eventually you'll come around to something that actually is interesting and convincing. Irrelevant? Patronizing? The issue is not at the individual level anyway. It is a systemic issue. We are discussing on a post from Nature called "Researchers value null results, but struggle to publishing them". Here's your systemic issue, raised in one of highest impact journals. > It's telling if someone isn't willing to do this. What does it tell you? That research is not the person's current job maybe? > I am talking about what is widely seen as failed research And this is my point. It shouldn't be. > you even call it that yourself Nope. At best I used quotes around the word "failing". I don't see this discussion progressing and surfacing interesting points anymore. You are disrespectful. Your points are subtly moving targets. You are sharing irrelevant advice to someone who doesn't need them. You are sharing irrelevant baseball and food comparisons (that I tried to adopt anyway). You are misrepresenting what I wrote. You don't really engage the actual topic. The whole discussion is looping. I believe you are trolling me. I tried to assume my feeling about this was wrong and gave you too much attention as a result. I should have stopped earlier. That'll teach me. If Nature can't convince you there's an issue that has absolutely nothing to do with me, I won't neither. I'm done. Bye! | | |
| ▲ | 9rx a day ago | parent [-] | | > I have already said why this is not an option. All I can see is that you said people can't find a compelling reason to do it. But that was already said long before you ever showed up and is specifically the point made in the comment you originally responded to... What do you think you are adding by just repeating that original comment over and over? > Recognized research doesn't currently happen in blog posts. Stands to reason. Who is doing it? Nobody is going to recognize something that doesn't exist! You have to demonstrate the value first. That is true in everything. Research is not somehow magically different. > What does it tell you? That nobody wants to do it. But what do you want to tell us? We already knew that nobody wants to do it. > I don't see this discussion progressing and surfacing interesting points anymore. It was never interesting, only humorous. Where did you find interest? > You are disrespectful. Ad hominem is a logical fallacy. > Your points are subtly moving targets. There is no apparent shift from my original comment as far as I can see. It is possible that you have misunderstood something, I suppose. I'm happy to keep trying to aid in your understanding. > You are sharing irrelevant advice to someone who doesn't need them. HN purportedly has 5 million monthly users. What makes you the expert on what they do and don't need? Get real. > You are misrepresenting what I wrote. It is possible, even likely, that I misunderstood what you wrote. But usually when you recognize that someone misunderstood you try to work with them in good faith to find an understanding, not run away crying that your precious words weren't written well enough to be understood, so I'm not sure you have really thought this through. > I believe you are trolling me. No you don't. The minute you legitimately thought I was a troll, you would have immediately cut off contact. Instead, you wrote a lengthy reply to give me your heartfelt goodbye. You can say this, but actions tell the true tale. Why make shit up? > If Nature can't convince you Said article in Nature effectively says the same things I have. What would it need to convince me of? It is on the very same page. I'm not sure why you keep thinking you aren't (even though you clearly are). Perhaps you've confused HN with Reddit and are trying to "win" some stupid "argument" nobody cares about — most especially me? That would explain why you keep repeating my comments in what appears to be some kind of "combative" way. > I'm done. Got it. This pretty much proves that is exactly what you were trying to do. What motivates this? |
|
|
|
|
|
|
|
|
| |
| ▲ | reliabilityguy a day ago | parent | prev [-] | | > It's easy to call it shameful when it isn't you who has to do the work. What work? The work of writing the paper? It seems to me that it’s better that research group A will spend 10 days to write the paper about their dead-end experiment, than 20 other research groups will do the same series of experiments, wasting a lot of time, money, and energy just because there was no paper that said “it doesn’t work”. Perhaps it would be better if those 20 research groups would instead try 20 different ways to fix the faulty method in hopes to get somewhere, than dosing the same thing. I do not understand how it’s even a question for debate. |
| |
| ▲ | 4 days ago | parent | prev [-] | | [deleted] |
|
