This post was originally published on here
A team of EPFL scientists tested the reproducibility of five decades of research on fly immunity. Most of those research results proved to be valid, but the team made one surprising discovery: the non-reproducible results were most often found in highly prestigious journals.
These are tough times for reproducibility. While this principle underpins the credibility of science, it has come under threat over the past 15 years – and researchers can’t do much about it. The reproducibility of research results is what ensures that a given finding can be obtained by another group of scientists working in a different environment. Yet in a range of fields – from biology to human behavior – a growing number of studies are proving to be non-reproducible. This is raising concerns about how reliable some of the scientific literature is and how the existing research system rewards certain practices.
To evaluate the extent of the problem, Bruno Lemaitre, an immunologist who heads EPFL’s Lemaitre Lab, launched the ReproSci project. He and his team spent six years working with other research groups to meticulously review, unpack and test the reproducibility of 400 research papers published in his field of drosophila immunity between 1959 and 2011. For each of these “cold cases” as he calls them (see interview below), the team distilled the authors’ scientific claims – identifying over 1,000 in all – and compared them with the results of subsequent studies on the same topics. For 45 key claims, no subsequent studies were found so the team reproduced the experiments themselves. The outcome of all this is available online (reprosci.epfl.ch) for the benefit of the entire scientific community.
A growing trend
The ReproSci findings have been written up in two articles which are now available on the bioRxiv server and are being reviewed for publication on eLife (see below). The project’s findings were mostly encouraging, as the team was able to confirm 61% of the scientific claims. “Only” 7% of the claims proved to be non-reproducible. Around 24% of the claims had never been verified, which is what the EPFL team did for some of them. And of these, a significant number couldn’t be reproduced, meaning that the actual percentage of non-reproducible claims is higher than 7% – probably between 15% and 18%. Taking all these figures into account, the research team concluded that a little over 80% of the published scientific claims were sound. This suggests that reproducibility may not be as big of a problem as many fear. Notwithstanding some non-reproducible findings, research results seemed to be generally reliable. However, the ReproSci project revealed an interesting incongruity: the unverifiable claims were often found where you would least expect them: in flagship journals such as Nature, Science and Cell and in papers written by scientists at prestigious institutes.
The second article provided further insight into these conclusions. The metastudy found no correlation between non-reproducibility and the authors’ experience or number of publications. Instead, the problem occurred most often in studies with researchers who came from outside the field or were involved in the study on an exploratory or opportunistic basis. Another key finding was that the percentage of non-reproducible claims tended to increase over time as the field became more popular and got more media coverage.
An impact on several fields
The ReproSci findings were consistent with those from other reproducibility studies. For instance, the Reproducibility Project: Cancer Biology initiative published by eLife found that nearly 50% of the claims in cancer studies appearing in top journals were not reproducible.
With ReproSci, Lemaitre has provided a rare snapshot of reproducibility across an entire life-science field, with quantitative conclusions and data made available in open access. What’s more, the conclusions are relevant not only to drosophila research. They show how natural human tendencies – such as wanting to boost visibility, raise funding and follow the latest research trends – can influence the reliability of a study’s results. By compiling evidence, shedding light on hidden aspects of the research system and calling on other scientists to get involved, the ReproSci team is getting people to think more broadly about what makes science trustworthy. Lemaitre’s approach goes beyond the technical aspects to remind scientists that reproducibility isn’t just about following the right protocol but should be an integral part of their culture and common practices.
Cover image: Original PR cover image.
Loading ad…
Stats
-
Recommendations n/a
n/a positive of 0 vote(s) - Views 51
- Comments 0
