Critical Assessment of Metagenome Interpretation – a benchmark of computational metagenomics software

By A. Sczyrba, P. Hofmann, Peter Belmann, D. Koslicki, Stefan Janssen, Johannes Dröge, Ivan Gregor, Stephan Majda, Jessika Fiedler, Eik Dahms and 57 more. 2017

675 citations recorded by the source corpus.

Methods for assembly, taxonomic profiling and binning are key to interpreting metagenome data, but a lack of consensus about benchmarking complicates performance assessment. The Critical Assessment of Metagenome Interpretation (CAMI) challenge has engaged the global developer community to benchmark their programs on highly complex and realistic data sets, generated from ∼700 newly sequenced microorganisms and ∼600 novel viruses and plasmids and representing common experimental setups. Assembly and genome binning programs performed well for species represented by individual genomes but were substantially affected by the presence of related strains. Taxonomic profiling and binning programs were proficient at high taxonomic ranks, with a notable performance decrease below family level. Parameter settings markedly affected performance, underscoring their importance for program reproducibility. The CAMI results highlight current challenges but also provide a roadmap for software selection to answer specific research questions.

Subfields: Ecology, Molecular Biology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Ecology

In Molecular Biology

Citations in the corpus

dRep: a tool for fast and accurate genomic comparisons that enables improved genome recovery from metagenomes through de-replication (2017, neutral): “However, co-assembly dramatically increases the data set size and complexity, especially when multiple different strains of the same species are present across the sample series, and can result in fragmented assemblies (Sczyrba et al., 2017).”
Optimizing taxonomic classification of marker-gene amplicon sequences with QIIME 2’s q2-feature-classifier plugin (2018, neutral): “20 phylum to family level but performance degrades at genus and species levels; different 416 methods display superior performance according to different performance metrics; and 417 parameter configuration dramatically impacts performance [26].”
IDseq—An open source cloud-based pipeline and analysis service for metagenomic pathogen detection and monitoring (2020, neutral): “Benchmarking of mNGS tools is a well-recognized challenge within the field [39, 59, 60] .”
Current challenges and best-practice protocols for microbiome analysis (2019, neutral): “Critical Assessment of Metagenomic Interpretation (CAMI)) [69, 70].”
IMG/M v.5.0: an integrated data management and comparative analysis system for microbial genomes and microbiomes (2018, background): “A myriad of binning tools and approaches are available (for a review, see (37)), with a recent assessment providing important benchmark comparisons across a subset of binning algorithms and tools (41).”
MetaWRAP—a flexible pipeline for genome-resolved metagenomic data analysis (2018, methodology): “While a number of sophisticated tools such as CONCOCT, Max-Bin, and metaBAT have been developed to address binning, it is still an actively improving field [9, 19 – 21].”
Integrating taxonomic, functional, and strain-level profiling of diverse microbial communities with bioBakery 3 (2020, methodology): “…(i) 49 synthetic metagenomes (10 Airways, 10 Gastrointestinal Tract, 10 Oral, 10 Skin, 9 Urogenital tract) provided by the 2nd CAMI challenge (Sczyrba et al., 2017) resemble the composition of the Human Microbiome as described by the Human Microbiome Project (Turnbaugh et al., 2007) ;…”
Robust taxonomic classification of uncharted microbial sequences and bins with CAT and BAT (2019, methodology): “First, it allows us to compare CAT to any of the taxonomic classifiers tested in the CAMI challenge (referred to as “taxonomic binners” in [22]).”
Species-level functional profiling of metagenomes and metatranscriptomes (2018, neutral): “This advantage follows from the known challenge of detecting low-coverage metagenomic sequences by assembly 31 , which was also observed in our synthetic evaluations.”