A prediction-based resampling method for estimating the number of clusters in a dataset

By Sandrine Dudoit, Jane Fridlyand. Genome Biology, 2002

813 citations recorded by the source corpus.

BackgroundMicroarray technology is increasingly being applied in biological and medical research to address a wide range of problems, such as the classification of tumors. An important statistical problem associated with tumor classification is the identification of new tumor classes using gene-expression profiles. Two essential aspects of this clustering problem are: to estimate the number of clusters, if any, in a dataset; and to allocate tumor samples to these clusters, and assess the confidence of cluster assignments for individual samples. Here we address the first of these problems.ResultsWe have developed a new prediction-based resampling method, Clest, to estimate the number of clusters in a dataset. The performance of the new and existing methods were compared using simulated data and gene-expression data from four recently published cancer microarray studies. Clest was generally found to be more accurate and robust than the six existing methods considered in the study.ConclusionsFocusing on prediction accuracy in conjunction with resampling produces accurate and robust estimates of the number of clusters.

Subfields: Artificial Intelligence, Molecular Biology

View Paper: https://www.semanticscholar.org/paper/bfb22cdbf8fc93050daa80839432b4661e9ab93b

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Molecular Biology

Citations in the corpus

Exploring the conditional coregulation of yeast gene expression through fuzzy k-means clustering (2002, methodology): “Because of the dependence of k-means clustering on k , a number of methods have been developed to estimate this parameter [16,17].”
Recovering the number of clusters in data sets with noise features using feature rescaling factors (2015, background): “Difficulties in the estimation of the number of clusters raised by the presence of noise features in a data set have been considered before [12], however, there is still a view that the issue raised by noise features deserves more consideration [7].”
Defining clusters from a hierarchical cluster tree: the Dynamic Tree Cut package for R (2008, background): “It remains an open research question how to choose optimal cutting parameters or how to estimate the number of clusters in the data set (Dudoit and Fridlyand, 2002).”
Bagging to Improve the Accuracy of A Clustering Procedure (2003, methodology): “In a related article (Dudoit and Fridlyand, 2002), the number of clusters estimated by the Clest procedure agreed with the ‘known’ number of clusters for both datasets.”
Cluster-wise assessment of cluster stability (2007, background): “Examples are Ben-Hur et al. (2002), Bryan (2004), Dudoit and Fridlyand (2002), Grün and Leisch (2004), Lange et al. (2004), Monti et al. (2001) and Tibshirani and Walther (2005).”
WGCNA: an R package for weighted correlation network analysis (2008, methodology): “Although the height and shape parameters of the Dynamic Tree Cut method provide improved exibility for branch cutting and module detection, it remains an open research question how to choose optimal cutting parameters or how to estimate the number of clusters in the data set [30].”
Classification Approaches in Neighborhood Research: Introduction and Review (2011, neutral): “The general term for such methods is cluster number estimation measures; they include among other approaches cross-validation techniques (cf. Dudoit and Fridlyand, 2002; Tibshirani and Walther, 2005; Fang et al., 2007) and cluster validity indices (Maulik and Bandyopadhyay, 2002; Kryszczuk and…”
Stability-Based Validation of Clustering Solutions (2004, methodology): “Ground-truth information on the correct number of clusters is available for both data sets that have also been re-analyzed by Dudoit and Fridlyand (2002).”
Statistical Significance of Clustering for High-Dimension, Low–Sample Size Data (2008, background): “(For more recent work in this area, see Tibshirani, Walther, and Hastie 2001; Dudoit and Fridlyand 2002; Sugar and James 2003.)”
Is My Network Module Preserved and Reproducible? (2011, methodology): “, the Rand index [53] or prediction based statistics [26,27].”
Consensus Clustering: A Resampling-Based Method for Class Discovery and Visualization of Gene Expression Microarray Data (2003, methodology): “The interested reader can find a more comprehensive survey of the topic in several recent books and papers (e.g., Dudoit & Fridlyand, 2002; Hastie, Tibshirani, & Friedman, 2001; Jain & Dubes, 1988).”
Inference of patient-specific pathway activities from multi-dimensional cancer genomics data using PARADIGM (2010, background): “Approaches for interpreting genome-wide cancer data have focused on identifying gene expression profiles that are highly correlated with a particular phenotype or disease state, and have led to promising results (Allison et al., 2006; Dudoit and Fridlyand, 2002; Tusher et al., 2001).”
Gene expression profiling spares early breast cancer patients from adjuvant therapy: derived and validated in two population-based cohorts (2005, methodology): “Class prediction using k genes was carried out using a diagonal linear discriminant analysis method [25], Table 1”