SCIENCE! Believe in Science? Bad Big-Data Studies May Shake Your Faith.

Today, the problem is not the scarcity of data, but the opposite. We have too much data, and it is undermining the credibility of science.

Luck is inherent in random trials. In a medical study, some patients may be healthier. In an agricultural study, some soil may be more fertile. In an educational study, some students may be more motivated. Researchers consequently calculate the probability (the p-value) that the outcomes might happen by chance. A low p-value indicates that the results cannot easily be attributed to the luck of the draw.

How low? In the 1920s, the great British statistician Ronald Fisher said that he considered p-values below 5% to be persuasive and, so, 5% became the hurdle for the “statistically significant” certification needed for publication, funding and fame.

It is not a difficult hurdle. Suppose that a hapless researcher calculates the correlations among hundreds of variables, blissfully unawarethat the data are all, in fact, random numbers. On average, one out of 20 correlations will be statistically significant, even though every correlation is nothing more than coincidence.

Real researchers don’t correlate random numbers but, all too often, they correlate what are essentially randomly chosen variables. This haphazard search for statistical significance even has a name: data mining. As with random numbers, the correlation between randomly chosen, unrelated variables has a 5% chance of being fortuitously statistically significant. Data mining can be augmented by manipulating, pruning and otherwise torturing the data to get low p-values.

Science still works. It’s just a shame so few “scientists” practice it.