Best Practices for Benchmarking Germline Small Variant Calls in Human Genomes

By P. Krusche, Len Trigg, Paul C. Boutros, Christopher E. Mason, F. M. De La Vega, Benjamin L. Moore, Mar Gonzàlez-Porta, M. Eberle, Z. Tezak, Samir Lababidi and 7 more. Nature Biotechnology, 2019

350 citations recorded by the source corpus.

Standardized benchmarking approaches are required to assess the accuracy of variants called from sequence data. Although variant-calling tools and the metrics used to assess their performance continue to improve, important challenges remain. Here, as part of the Global Alliance for Genomics and Health (GA4GH), we present a benchmarking framework for variant calling. We provide guidance on how to match variant calls with different representations, define standard performance metrics, and stratify performance by variant type and genome context. We describe limitations of high-confidence calls and regions that can be used as truth sets (for example, single-nucleotide variant concordance of two methods is 99.7% inside versus 76.5% outside high-confidence regions). Our web-based app enables comparison of variant calls against truth sets to obtain a standardized performance report. Our approach has been piloted in the PrecisionFDA variant-calling challenges to identify the best-in-class variant-calling methods within high-confidence regions. Finally, we recommend a set of best practices for using our tools and evaluating the results. A new standard allows the accuracy of variant calls to be assessed and compared across different technologies, variant types and genomic regions.

Subfields: Genetics, Molecular Biology

View Paper: https://www.semanticscholar.org/paper/55cd83b454e084a6e1a2c8db8ce5a6cf00183a09

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Genetics

In Molecular Biology

Citations in the corpus

Merqury: reference-free quality, completeness, and phasing assessment for genome assemblies (2020, background): “…the highly variable major histocompatibility complex (MHC) is excluded from the Genome in a Bottle (GIAB) and Global Alliance for Genomic Health (GA4GH) reference panels [26] due to its repetitive nature and need of a specialized mapping strategy to account for the high allelic diversity [29].”
Accurate circular consensus long-read sequencing improves variant detection and assembly of a human genome (2019, background): “Evaluation of variant calls and assemblies relies on high-quality benchmarks and supporting tools 47 .”
A draft human pangenome reference (2022, methodology): “The comparison between the callsets and truth set was made with RTG’s vcfeval (Cleary et al., 2015) and Illumina’s hap.py tool (Krusche et al., 2019) on confident regions of the benchmark.”
An open resource for accurately benchmarking small variant and reference calls (2019, methodology): “10 The broadly consented and available GIAB samples from PGP have at least three strengths relative to the other samples: (1) we can continue to sequence these renewable cell lines with new technologies and improve our benchmark set over time, (2) any clinical or research laboratory can sequence…”
Best practices for variant calling in clinical sequencing (2020, methodology): “Software packages such as BCFtools make it possible to merge and reconcile multiple variant callsets (in VCF format) into one, though care should be taken to properly handle complex variants and/ or differences in variant representation [70].”