Initialization is critical for preserving global data structure in both t-SNE and UMAP

By Dmitry Kobak, George C. Linderman. 2021

318 citations recorded by the source corpus.

1Institute for Ophthalmic Research, University of Tübingen, Tübingen, Germany. 2Applied Mathematics Program, Yale University, New Haven, CT, USA. ✉e-mail: dmitry.kobak@uni-tuebingen.de; george.linderman@yale.edu One of the most ubiquitous analysis tools in single-cell transcriptomics and cytometry is t-distributed stochastic neighbor embedding (t-SNE)1, which is used to visualize individual cells as points on a two-dimensional scatterplot such that similar cells are positioned close together2. A related algorithm, called uniform manifold approximation and projection (UMAP)3, has attracted substantial attention in the single-cell community4. In Nature Biotechnology, Becht et al.4 argued that UMAP is preferable to t-SNE because it better preserves the global structure of the data and is more consistent across runs. Here we show that this alleged superiority of UMAP can be entirely attributed to different choices of initialization in the implementations used by Becht et al.: the t-SNE implementations by default used random initialization, while the UMAP implementation used a technique called Laplacian eigenmaps (LE)5 to initialize the embedding. We show that UMAP with random initialization preserves global structure as poorly as t-SNE with random initialization, while t-SNE with informative initialization performs as well as UMAP with informative initialization. On the basis of these observations, we argue that there is currently no evidence that the UMAP algorithm per se has any advantage over t-SNE in terms of preserving global structure. We also contend that these algorithms should always use informative initialization by default. At the core of both t-SNE and UMAP are loss functions that make similar points attract each other and push dissimilar points away from each other. Both algorithms minimize their loss functions by using gradient descent. Gradient descent begins with some initial configuration of points, and with each iteration the points are moved to decrease the loss function. The specific implementations of these algorithms used by Becht et al. differed in how the initial configuration of points was chosen: the t-SNE implementations placed the points randomly, whereas the UMAP implementation used LE5, an algorithm that can often achieve globally accurate embedding on its own6. The effect of this difference on how well the two algorithms preserve global structure was not discussed or investigated by Becht et al. We can illustrate the importance of initialization for both algorithms using a simple toy dataset (Fig. 1). We sampled n = 7,000 points from a circle with some added Gaussian noise and used UMAP and t-SNE to construct an embedding. We kept all parameters for both algorithms at their default values and only changed the initialization. The t-SNE algorithm with random initialization produced a knot; UMAP with random initialization produced a tangled web with multiple tears. In both cases, the global structure was evidently not preserved; indeed, gradient descent in t-SNE and UMAP only pulls close neighbors together and is not much influenced by the global arrangement of points. At the same time, LE recovers the original circle for this toy dataset and, when used for initialization, strongly improves the UMAP result. A method often recommended for initialization in t-SNE is principal component analysis (PCA)2. Here PCA also recovers the original circle and strongly improves the t-SNE result. In both cases, only with informative initialization can UMAP and t-SNE produce a faithful representation of the closed one-dimensional manifold—the circle. Using the code published by Becht et al., we analyzed the separate effects of initialization and algorithm on their results by adding UMAP with random initialization and t-SNE (using FIt-SNE7) with PCA initialization to the benchmarking comparison. Apart from the initialization, both algorithms were run with the same parameters as in Becht et al. We used all three datasets analyzed in the original publication (sample sizes from 320,000 to 820,000 cells)8–10. To quantify preservation of global structure, Becht et al. computed Pearson correlation between pairwise Euclidean distances in high-dimensional space and in the embedding. To quantify the reproducibility of the embedding, the authors embedded random subsamples of the data and measured the correlation of the coordinates of subsample embeddings with the coordinates of the full-dataset embeddings (up to symmetries around the coordinate axes). Our results show that t-SNE and UMAP with random initialization perform similarly poorly with regard to both metrics, whereas t-SNE and UMAP with PCA and LE initialization, respectively, perform similarly well (Table 1). See Extended Data Figs. 1–6 for the exact analogs of the original figures from Becht et al. Becht et al. wrote that their findings were “consistent with the idea” that UMAP performs “optimizations that are sensitive to global features of the data, thus reaching similar arrangements more consistently.” Our results show that this conclusion can be misleading: their findings were in fact not due to UMAP optimizations but rather to its initialization. We have recently argued that, for single-cell transcriptomic data, t-SNE with PCA initialization produces more meaningful embeddings than t-SNE with random initialization2. The findings of Becht et al., when interpreted correctly, also underscore the importance of using informative initialization and suggest that it should be used as the default option in t-SNE and UMAP implementations. PCA initialization has always been the default in openTSNE11, a Python reimplementation of FIt-SNE, and FIt-SNE v.1.2 now also uses it by default. OpenTSNE v.0.4 now also supports LE initialization. Importantly, t-SNE with non-random initialization should not be considered a new algorithm or even an extension of the original t-SNE; it is exactly the same algorithm with the same loss function, and almost any existing implementation trivially allows the use of any given initialization, including the PCA-based one. Initialization is critical for preserving global data structure in both t-SNE and UMAP

Subfields: Computer Networks and Communications, Electrical and Electronic Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Electrical and Electronic Engineering

Citations in the corpus

Understanding How Dimension Reduction Tools Work: An Empirical Approach to Deciphering t-SNE, UMAP, TriMAP, and PaCMAP for Data Visualization (2020, background): “However, just as much recent research has discovered (Kobak and Linderman, 2021; Kobak and Berens, 2019), we found that the initialization has a much larger influence on the success rate than simply faster convergence”
openTSNE: a modular Python library for t-SNE dimensionality reduction and embedding (2019, background): “Kobak and Linderman (2021) showed that the initialization of force-directed layout algorithms, such as t-SNE, UMAP (McInnes et al. 2018), and ForceAtlas2 (Jacomy, Venturini, Heymann, and Bastian 2014), largely dictates the global consistency of resulting embeddings.”
SCEHR: Supervised Contrastive Learning for Clinical Risk Prediction using Electronic Health Records (2021, methodology): “The t-SNE is initialized by PCA as suggested in [38].”
Parametric UMAP Embeddings for Representation and Semisupervised Learning (2020, methodology): “…while global algorithms, like multidimensional scaling (MDS), attempt to preserve all relationships during embedding. e d An open issue in dimensionality reduction is how to balance local and global structure preservation (Becht et al., 2019; De Silva & Tenenbaum, 2003; Kobak & Linderman, 2021).”