A comparison of automatic cell identification methods for single-cell RNA sequencing data

By T. Abdelaal, Lieke Michielsen, D. Cats, Dylan Hoogduin, H. Mei, Marcel J. T. Reinders, A. Mahfouz. Genome Biology, 2019

517 citations recorded by the source corpus.

Single-cell transcriptomics is rapidly advancing our understanding of the cellular composition of complex tissues and organisms. A major limitation in most analysis pipelines is the reliance on manual annotations to determine cell identities, which are time-consuming and irreproducible. The exponential growth in the number of cells and samples has prompted the adaptation and development of supervised classification methods for automatic cell identification. Here, we benchmarked 22 classification methods that automatically assign cell identities including single-cell-specific and general-purpose classifiers. The performance of the methods is evaluated using 27 publicly available single-cell RNA sequencing datasets of different sizes, technologies, species, and levels of complexity. We use 2 experimental setups to evaluate the performance of each method for within dataset predictions (intra-dataset) and across datasets (inter-dataset) based on accuracy, percentage of unclassified cells, and computation time. We further evaluate the methods’ sensitivity to the input features, number of cells per population, and their performance across different annotation levels and datasets. We find that most classifiers perform well on a variety of datasets with decreased accuracy for complex datasets with overlapping classes or deep annotations. The general-purpose support vector machine classifier has overall the best performance across the different experiments. We present a comprehensive evaluation of automatic cell identification methods for single-cell RNA sequencing data. All the code used for the evaluation is available on GitHub (https://github.com/tabdelaal/scRNAseq_Benchmark). Additionally, we provide a Snakemake workflow to facilitate the benchmarking and to support the extension of new methods and new datasets.

Subfields: Biophysics, Molecular Biology

View Paper: https://www.semanticscholar.org/paper/d81bdaf1abed1682e6eaeed465f9b2b50d3b441c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biophysics

In Molecular Biology

Citations in the corpus

scDeepSort: a pre-trained cell-type annotation method for single-cell transcriptomics using deep learning with a weighted graph neural network (2021, methodology): “Interestingly, it seems easy to map cells across the human pancreas as reported in the recent benchmark analysis (18), wherein the widely-used human pancreas datasets (37,42– 44) were tested.”
Best practices for single-cell analysis across modalities (2023, methodology): “Generally, pseudobulk methods with sum aggregation and mixed models such as MAST with random effect setting were found to be superior to naive methods, such as the popular Wilcoxon rank-sum test, which does not account for within-sample correlation 88 .”
Fully-automated and ultra-fast cell-type identification using specific marker combinations from single-cell transcriptomic data (2022, methodology): “Furthermore, supervised methods are notoriously time-consuming to train, as well as error prone to technical artifacts in the training data, which affect their prediction ability for new scRNA-seq data 47 .”