Front-line victims of the replication crisis have been identified by New Zealand and American scientists: studies themselves.
Colin Camerer, Brian Nosek and their colleagues from Massey University, America’s Center for Open Science, and Caltech, attempted to reproduce the findings of 21 social science studies, published in the journals Nature and Science between 2010 and 2015. On average, only a quarter from Nature were replicable, while just two in five from Science could be reaffirmed.
Ironically, they used very small sample sizes (only reproducing four studies in Nature and 17 in Science) – a frequent reason for studies being unreplicable. Yet they attempted to counter this by, in their replication studies, using sample sizes that were five times larger than the ones used in the original studies.
So why were so many of the original studies flawed? Camerer and his colleagues said false positives and inflated effect sizes could be culprits. Yet they also implicitly blamed the original researchers for this. After surveying 400 researchers, they found a strong correlation between replicability and beliefs about it. This led them to conclude that the researchers were likely aware that their studies were not reproducible.
That they published anyway could be symptomatic of the publish or perish culture that has infected academia.
Nevertheless, the replication study authors appeared disappointed with their results, given they pertained to premium journals “where one might expect greater editorial scrutiny”.
Nature Human Behaviour published their findings.
In an accompanying editorial, Scottish neurology professor Malcolm MacLeod took a more sanguine approach, where he extolled the virtues of ‘failed’ reproducibility. He said it fosters “great research opportunities, and pursuit of these opportunities is likely to lead to improved and better considered research designs”.
“For these reasons, current issues of the reproducibility of research should not be seen as a crisis, but as an opportunity.”
Email [email protected]