We Need a Journal of Null Results in every Field!

April 4th, 2012

I’ve been thinking more about some practical problems we’ve been developing in science.  Now, they aren’t as bad in my field of astronomy, since we’re relatively small, don’t involve huge sums of money, and rarely have results that are politically problematic.  But we still have the same problem:  null results don’t get published very often.

What I mean is that I have an idea, go do an observation or study, and find out that the effect I thought might be there isn’t.  And I don’t publish that null result because it isn’t very interesting usually, won’t get me a lot of citations, will take a lot of time, and may have a problem getting through a referee/editor who doesn’t feel that it’s very important.

But it is to the field!

I imagine that some of the things I’ve looked for, others have looked for.  Maybe a lot of them.  Maybe every quasar expert out there has had the same hypothesis and looked for a correlation of x vs. y to support a particular idea, and not found it.  Maybe if the first expert had published, it would have saved a lot of people a lot of time.

Or, also importantly, a lot of people look for an effect.  Say, 100 researchers.  One of them finds a significant result at the 3 sigma level, which would only happen a little less often than 1 time in 100 tries.  And that one “lucky” researcher publishes the result, misguiding an entire subfield, perhaps, who doesn’t know about the other 99 results that didn’t find a significance level so high.  I suspect this thing can be very important in difficult drug studies where companies testing their drugs only publish and submit the studies that show a positive effect and ignore them ones that don’t.

What I’m suggesting is a journal, or venue such as an on-line clearinghouse, where scientists can write abstracts or non-refereed papers describing their null results.  Right now they end up in personal journals and ignored or forgotten, doomed to be repeated or their absence to contribute to a misleading view.

I’ve got one null result that has set on my desk for about five years, following a referee who missed the point and wanted a lot of work concerning an unimportant side issue (really, I’ve had enough time to look past my biases on the project).  It should be out there.  It isn’t.  It hasn’t gotten back to the top of the list in five years because I have other more promising projects to do.  It’s a shame.  Maybe I could have worked harder, but I literally have 30 projects on my to do list now (not just things to do, but projects that could all end up as papers).

If we need a Journal of Null Results in astronomy, we for sure need one in about every other field of science you can imagine!

For more reading on this topic, check out Wrong: Why experts* keep failing us–and how to know when not to trust them *Scientists, finance wizards, doctors, relationship gurus, celebrity CEOs, … consultants, health officials and more by David H. Freedman.

Share/Bookmark

Tags: ,

You can follow any responses to this entry through the RSS 2.0 feed. You can skip to the end and leave a response. Pinging is currently not allowed.