Estimating the number of clusters in a dataset via the gap statistic

By Robert Tibshirani. 2000

5,218 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Detecting Backdoor Attacks on Deep Neural Networks by Activation Clustering (2018, methodology): “We also experimented with using the gap statistic (Tibshirani et al., 2001) to compare the relative fit of one versus two clusters but this was largely unsuccessful.”
A tutorial on spectral clustering (2007, methodology): “…as the ratio of within-cluster and between-cluster similarities, over information-theoretic criteria (Still and Bialek, 2004), the gap statistic (Tibshirani, Walther, and Hastie, 2001), to stability approaches (Ben-Hur, Elisseeﬀ, and Guyon, 2002; Lange, Roth, Braun, and Buhmann, 2004;…”
Differentially Private K-Means Clustering (2015, background): “Several heuristics and statistics [29, 32] have been proposed to determine the number of clusters k automatically.”
NbClust: An R Package for Determining the Relevant Number of Clusters in a Data Set (2014, methodology): “The estimated Gap statistic proposed by Tibshirani et al. (2001) is computed using Equation 27. where B is the number of reference data sets generated using uniform prescription (Tibshirani et al. 2001) and W qb is the within-dispersion matrix deﬁned as in the Hartigan index.”
Revisiting k-means: New Algorithms via Bayesian Nonparametrics (2011, methodology): “s assume a ﬁxed, known value for kwhich, in many cases, is simply not known a priori. Some work has gone into the development of techniques to overcome this problem in the context of k-means, such as [2, 7, 8, 12, 14, 19, 21], which utilize various model selection criteria. In the case of mixture mo”
Extending the Akaike Information Criterion to Mixture Regression Models (2007, methodology): “Presently, users can apply extant approaches (e.g., Biernacki, Celeux, and Govaert 2000; Tibshirani, Walther, and Hastie 2001; Tadesse, Sha, and Vannucci 2005) to determine the number of components (but not variables) or information criteria, such as the Akaike information criterion (AIC) (Akaike…”
Understanding of Internal Clustering Validation Measures (2010, unknown): “literature [17] [18] [19] [20].”
K-means clustering: a half-century synthesis. (2006, background): “However, the generalization of the results provided by Milligan and Cooper (1985) and others (such as Tibshirani et al., 2001) has to be taken in the context of the Monte Carlo procedure implemented and the data generation process used.”
Finding the Number of Clusters in a Dataset (2003, methodology): “…are a Gaussian model–based approach using approximate Bayes factors (Kass and Raftery 1995; Frayley and Raftery 1998) and the gap statistic that compares the change in within-cluster dispersion with that expected under an appropriate null distribution (Tibshirani, Walther, and Hastie 2001).”
FLDetector: Defending Federated Learning Against Model Poisoning Attacks via Detecting Malicious Clients (2022, methodology): “Specifically, we cluster the clients based on their suspicious scores s 1, s t 2, · · · , s t n , and we use the Gap statistics [19] to determine the number of clusters.”
Ant-based and swarm-based clustering (2007, methodology): “Ant-based clustering has the advantage of automatically determining the number of clusters, and it was shown to outperform a statistical method (the Gap statistic, Tibshirani et al. 2001) in this respect (Handl et al. 2006).”
Poor-prognosis colon cancer is defined by a molecularly distinct subtype and develops from serrated precursor lesions (2013, methodology): “To further confirm this, we computed Gap statistic[37] for k = 1 to 5, and a peak was consistently found at k = 3 (Fig.”
Statistical power for cluster analysis (2020, background): “In fact, the popular “gap statistic” evaluation metric is explicitly designed for well-separated and non-overlapping clusters [21].”
Selection of K in K-means clustering (2005, background): “The f ( K ) plot for this data set suggests correctly that, when K ¼ 1, the clustering result is the most suitable for this data set.”
An Evolutionary Approach to Multiobjective Clustering (2007, methodology): “’s Gap statistic [57] can be considered a single-objective approach to model selection, as it takes only a single clustering objective into account.”