What is Reproducibility?

By Hidaya Aliouche, B.Sc.Reviewed by Emily Henderson, B.Sc.

Introduction
Defining Reproducibility
Measuring Precisely
Reproducible Research
Factors Affecting Reproducibility in Life Science Research
The Reproducibility Problem
Factors That Contributed to the Lack of Reproducibility
References

Reproducibility refers to the consistency of measurements. It is the extent to which a tool can produce the same result when used repeatedly under the same circumstances. Reproducibility is used interchangeably with the terms repeatability and reliability.

A measurement can have good reproducibility while demonstrating poor validity. The converse is not true; a measurement that has good validity cannot have good reproducibility.

Image Credit: Gorodenkoff/Shutterstock.com

Defining Reproducibility

Different scientific disciplines and institutes use the words replicability and reproducibility inconsistently - or in some cases - contradicting one another. This creates difficulties in assessing reproducibility due to an absence of a standard definition for this term.

However, in the Biological Sciences, reproducibility can be defined as obtaining consistent results using the same input variables, methodological and computational steps, and analysis conditions. The term reproducibility is closely related to replicability, which is the process of obtaining consistent results across studies that answered the same scientific inquiry, each of which has obtained its data.

According to the American Society for cell biology, reproducibility can be defined using a multi-tiered approach, which reflects how the term is perceived throughout different scientific communities.

1. Direct replication: attempts to reproduce a previously observed result by using the same experimental design and conditions as defined in the original study
2. Analytic replication: reproducing a series of scientific findings through a re-analysis of the original data set
3. Systemic replication: reproducing an experimental finding using different experimental conditions (for example, in a different animal model or cell culture system)
4. Conceptual replication: evaluation of data validity using a different set of experimental conditions or methods

Measuring Precisely

All forms of scientific observation, counting, measuring, or both, are involved. Scientific measurements are broad and can consist of different types: temperature, colorimetric properties, electromagnetic properties, time, spatial dimensions, electric current, material properties, acidity, and concentration. These examples are not exclusive and only offer a small example from the natural sciences. For example, scientific observations across other disciplines in the social sciences are similarly replete with various counts and measures.

Each measure has an associated margin of doubt or error; this reflects on certainty. All measurements, counts, and other forms of quantification are associated with uncertainties, which is a core feature of scientific measures.

Precision refers to the degree of closeness in measurement, which narrows the margin of error associated with the scientific observation. The degree of closeness is related to the scale at which an observation is being measured. For example, measurement of distances ranges from meters to centimeters to millimeters, to several orders of descending magnitude – to microns, nanometers, and angstroms. With shrinking size comes an increase in exactness, and the proximity of one measurement to another can be determined more precisely.

Precision is also distinct from the accuracy of a measuring tool and can be illustrated as follows:

In the first case, the three dots are in the outer ring, separate from one another and not close to the bullseye, illustrating low accuracy and precision. In the second scenario, the dots are all clustered in a tight band in an outer ring, which illustrates high precision but low accuracy. In the third scenario, the dots are located close to the bullseye but not close to one another, illustrating high accuracy but low precision.

Finally, in the fourth scenario, the dots are close to the bullseye's intended destination. In this final rendition, the outcome is highly accurate and precise. Visual inspections of means and standard deviations can reveal a data set's reproducible.

Reproducible Research

Reproducible research is a central tenant to open science. Using this methodology, a detailed description of the method used to obtain the data is provided so that it is easily accessible and can be reproduced. For research projects to be computationally reproducible, all data and files must be clearly labeled, separated, and documented. However, reproducibility continues to be a challenge in science as methods are difficult to obtain.

Factors Affecting Reproducibility in Life Science Research

The independent verification of data is a core tenant of scientific research across all forms of disciplines. To self-correct, researchers must be able to reproduce the findings of published studies to build on both the rigor of existing data and the body of existing work. Reproducibility is not necessarily a measure of ensuring correct results but ensuring the transparency of the methods enacted to obtain them in any given line of research.

Ideally, researchers should be able to recreate experiments, generate the same results, and arrive at the same conclusions. In practice, this is not always the case; biomedical research, for example, cannot be reproduced owing to the considerable variation of each of the methodologies employed. This throws into question the credibility of scientific findings, and despite heightened attention on the matter, there remains a lack of awareness among research trainees and students about the lack of reproducibility.

The Reproducibility Problem

According to a 2016 nature survey, over 70% of researchers in the field of biology were found to be unable to reproduce the findings of other scientists. Moreover, ~ 60% of researchers could not reproduce their findings. This poor reproducibility results in lower scientific output efficiency, slows scientific progress, and increases the amount of time and money wasted, culminating in reduced confidence in the public trust in scientific research.

Indeed, the cost of non-reproducible research is estimated at $28 billion per year( that is, the amount of money spent on preclinical research that cannot be reproduced).

Factors That Contributed to the Lack of Reproducibility

There is no one defining cause of lack of reproducibility; instead, several categories can explain why research cannot be reproduced. These include:

• A lack of access to methodological details, raw data, and research materials used: in the absence of protocols, original data, and key research materials, reproduction is prevented, and researchers must devise new protocols to repeat the previous work. To improve the sharing of information, the systems and mechanisms for sharing unpublished data and research materials need to be made more robust to prevent sharing from being a hindrance to reproducibility
• Use of cell lines and microorganisms that have been misidentified, cross-contaminated, or over-passaged (over-transfer of cells from a previous culture into fresh growth medium): complicated and/or invalidated biological materials that aren't capable of being traced to their origin are not thoroughly authenticated or maintained, which therefore affects the ability to reproduce data. As a result of improperly maintaining biological materials, in combination with serial passing, the phenotype and genotype of the material are compromised, making reproducing data difficult
• Inability to manage complex data sets: in the absence of a core set of experimental parameters, poorly designed studies with a methodology that is not reported are likely to be incapable of reproducing. This is ultimately down to poor practices and reporting research results as well as poor experimental design
• Cognitive bias: impartiality is part of the scientific method; however, individuals' subjective social context affects the conduct of research. The biases, confirmation, selection, and reporting bias have been identified, alongside the bandwagon effect and cluster illusion. Confirmation bias is the act of unconsciously interpreting new evidence in ways that confirm existing beliefs or theories. This subsequently affects how information is obtained, analyzed, and recalled. Selection bias results in data selection for analysis that has not been properly randomized; consequently, the sample obtained is not representative of the whole population. The bandwagon effect describes a tendency to agree with a particular viewpoint without robust evaluation. This is done to maintain group harmony and may lead to accepting unproven theories that have become popular. Cluster illusion occurs when a pool of random data is thought to contain patterns when no actual patterns have arisen. This is based on the tendency of the brain to seek them out. Reporting bias occurs when there is a selective reporting of information from participants. This is a subconscious effect and may lead to underreporting of negative or undesirable data
• A competitive culture that rewards novel findings and penalizes negative results: the research system is built on the rapid publication of novel results. There is often an incentive for publishing novel findings and associated punishments for publishing negative results (i.e., data that shows no correlation). In many cases, hiring and promotion criteria in academic institutes emphasize the requirement to publish in high-impact journals; this is augmented by a competitive environment for research grants which culminate in incentivizing research to limit by reporting of successful methodologies that make experiments work better

Overall, reproducibility is essential to produce robust incredible research to promote advances in science. Several factors have been identified as impediments to reproducibility in life sciences. Accordingly, several guidelines and recommendations have emerged, but the practical implementation of these practices is more challenging.

To ensure reproducibility in the future, the scientific community must take an objective approach to design experiments and be willing to depict and publish their results accurately and thoroughly, with precise descriptions of all methodologies used.

In addition, publishers, funders, and policymakers are responsible for highlighting the lack of reproducibility and promoting better research practices and incentives in the life sciences. Ultimately this can help improve research practices and ensure the robust credibility of scientific data.

References:

• National Academies of Sciences, Engineering, and Medicine; Policy and Global Affairs; Committee on Science, Engineering, Medicine, and Public Policy; Board on Research Data and Information; Division on Engineering and Physical Sciences; Committee on Applied and Theoretical Statistics; Board on Mathematical Sciences and Analytics; Division on Earth and Life Studies; Nuclear and Radiation Studies Board; Division of Behavioral and Social Sciences and Education; Committee on National Statistics; Board on Behavioral, Cognitive, and Sensory Sciences; Committee on Reproducibility and Replicability in Science. Reproducibility and Replicability in Science. Washington (DC): National Academies Press (US); 2019 May 7. 3, Understanding Reproducibility and Replicability. Available from: https://www.ncbi.nlm.nih.gov/books/NBK547546/.
• Nature portfolio. Six factors affecting reproducibility in life science research and how to handle them. Available at: https://www.nature.com/articles/d42473-019-00004-y. Last accessed December 2021.
• Weintraub PG. (2016) The Importance of Publishing Negative Results. J Insect Sci. doi: 10.1093/jisesa/iew092.
• Munafò MR, Nosek B, Bishop D, et al. A manifesto for reproducible science. Nature Human Behaviour. doi: 10.1038/s41562-016-0021.
• Resnik DB, Shamoo AE. (2017) Reproducibility and Research Integrity. Account Res. doi: 10.1080/08989621.2016.1257387.

Further Reading

• All Research Content
• What is the Replication Crisis?
• What are the Advantages of Open-Access Research?
• Human Health Solutions through Animal Research