I have been meaning to write this post for a really long time, but it is one that I actually have to think about, so it has taken me this long to actually sit down and write it.
When you do a scientific study, you run your data through a calculation to figure out a p-value.Â A p-value tells you how likely it is that your data results are true and not based on coincidence.Â A p-value of .05 is usually considered safe and significant.Â That means that if you do this study one hundred times, you will get the same results ninety-five times.Â The lower your p-value, the better your results likely are.Â A p-value of .01 is usually considered quite good.Â (We were required to hit at least .05 when we did studies in college but our professor was always pushing for .01.)
Let’s say that you have three friends.Â You’d expect you and your three friends to have random birthdays spread out all over the year.Â But because there are so many people all over the world who also have three friends, some of those people are going to end up with all three friends who have birthdays on the very same day.Â Those are times when we say “what a coincidence” and move on.
Now, studies that actually find something are more likely to be published in journals and picked up by the mainstream.Â So, let’s say that you are doing a study on toddlers and napping.Â Your hypothesis is that toddlers that nap for at least two hours per day are much more likely to get into an Ivy League University.Â You are much more likely to get “Toddlers that Nap Are More Likely To Get Into Harvard” published.Â If your results are “There is No Connection Between Napping and Ivy League Acceptance”, probably no one is going to want to publish it.Â Which doesn’t mean that your study is any less correct and valuable, it just means that it is less likely to see the light of day.
Now, p-values and the positive results bias in publishing cause a problem.Â If one hundred people do the napping versus Harvard study, ninety-five of them are going to find nothing.Â But five of them will find something, based on coincidence.Â And those five are more likely to get published.Â So, when a new study comes out, you always have to be aware of the fact that it isn’t necessarily true.
A recent one I’ve heard about that strikes me as a potential example of this positive effect bias is the safety of ultrasounds study which found higher rates of left handed boys.Â I haven’t done a whole lot of reading on this, but when you read the studies, there is a lot of “the effect was only found in these extremely specific cases” and “these results have not been replicated”.
Probably the best example of the positive bias in publishing is when two scientists claimed that cold-fusion ought to be possible.Â This discovery got a lot of publicity because, you know, awesome, cheap and easy energy!Â But then a lot of other scientists came out and said “wait a minute, my data said that cold fusion isn’t possible, I mean, I didn’t try to get it published because no one wants to read about how it isn’t possible.”Â So in this case, the results were discovered to be due to probability, but in a lot of other studies, it may never be discovered.
The best way to feel confident that a study is true is to see that it has been replicated.Â You can just use your common sense too.Â If someone tells you that a study found that wearing a bra causes breast cancer and that just doesn’t sound right to you, you can look at the study and see why the results are likely due to chance or to correlation.
And that is why you can’t even trust science.