Measuring reproducibility of high-throughput experiments

By Qunhua Li, James B. Brown, Haiyan Huang, Peter J. Bickel. 2011

1,082 citations recorded by the source corpus.

Reproducibility is essential to reliable scientific discovery in high-throughput experiments. In this work we propose a unified approach to measure the reproducibility of findings identified from replicate experiments and identify putative discoveries using reproducibility. Unlike the usual scalar measures of reproducibility, our approach creates a curve, which quantitatively assesses when the findings are no longer consistent across replicates. Our curve is fitted by a copula mixture model, from which we derive a quantitative reproducibility score, which we call the "irreproducible discovery rate" (IDR) analogous to the FDR. This score can be computed at each set of paired replicate ranks and permits the principled setting of thresholds both for assessing reproducibility and combining replicates. Since our approach permits an arbitrary scale for each replicate, it provides useful descriptive measures in a wide variety of situations to be explored. We study the performance of the algorithm using simulations and give a heuristic analysis of its theoretical properties. We demonstrate the effectiveness of our method in a ChIP-seq experiment.

Subfields: Biophysics, Information Systems and Management, Materials Chemistry

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biophysics

In Information Systems and Management

In Materials Chemistry

Citations in the corpus

Iterative random forests to discover predictive and stable high-order interactions (2017, neutral): “, 1984) to a large extent mitigates normalization issues that are a major concern in the analysis of genomics data, where signal-to-noise ratios vary widely even between biological replicates (Landt et al., 2012; Li et al., 2011).”
A deep generative model for single-cell RNA sequencing with application to detecting differentially expressed genes (2017, methodology): “To quantify this, we model the relationship between significance ranks using the Irreproducible Discovery Rate model for matched rank lists [25] and report correlation score of the reproducible components in Figure 2.”
ChIP-seq guidelines and practices of the ENCODE and modENCODE consortia (2012, methodology): “The irreproducible discovery rate (IDR) analysis methodology (Li et al. 2011) is now used to assess replicate agreement and set thresholds (discussed further below).”
An Integrated Encyclopedia of DNA Elements in the Human Genome (2012, methodology): “In addition we developed the irreproducible discovery rate (IDR) 13 measure to provide a robust and conservative estimate of the threshold where two ranked lists of results from biological replicates no longer agree ( i.e. , are irreproducible) and we applied this to defining sets of discrete…”
Bayesian Inference for a Generative Model of Transcriptome Profiles from Single-cell RNA Sequencing (2018, methodology): “(a, b) Evaluation of consistency with the irreproducible discovery rate (IDR) [44] framework (blue) and using AUROC (green) is shown for comparisons of B cells vs Dendritic cells (a) and CD4 vs CD8 T cells (b).”
ENCODE Data in the UCSC Genome Browser: year 5 update (2012, methodology): “com/p/align2rawsignal), peak calls from the SPP (15) and PeakSeq (16) peak callers filtered using the irreproducible discovery rate (17), computed RNA contigs from seven cellular localizations (18) and genome segmentations for the six Tier 1 and 2 cell lines.”
SeqGL Identifies Context-Dependent Binding Signals in Genome-Wide Regulatory Element Maps (2015, methodology): “We use IDR with a cutoff of 0.01 to identify reproducible subpeaks in each cell type.”
A deep generative model for gene expression profiles from single-cell RNA sequencing (2017, methodology): “To quantify this, we model the relationship between significance ranks using the Irreproducible Discovery Rate model for matched rank lists [25] and report correlation score of the reproducible components in Figure 2.”