Deep Networks With Large Output Spaces

By Sudheendra Vijayanarasimhan, Jonathon Shlens, Rajat Monga, Jay Yagnik. 2014

58 citations recorded by the source corpus.

Deep neural networks have been extremely successful at various image, speech, video recognition tasks because of their ability to model deep structures within the data. However, they are still prohibitively expensive to train and apply for problems containing millions of classes in the output layer. Based on the observation that the key computation common to most neural network layers is a vector/matrix product, we propose a fast locality-sensitive hashing technique to approximate the actual dot product enabling us to scale up the training and inference to millions of output classes. We evaluate our technique on three diverse large-scale recognition tasks and show that our approach can train large-scale models at a faster rate (in terms of steps/total time) compared to baseline methods.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Large Memory Layers with Product Keys (2019, background): “Another model is proposed in [43], where the authors develop a fast locality-sensitive hashing technique to approximate the dot product between large matrices and vectors in neural networks.”
ImageNet-21K Pretraining for the Masses (2021, background): “There are also works on ImageNet-21K that did not focus on pretraining: [53] used extra (unlabled) data from ImageNet-21K to improve knowledge-distillation training on ImageNet-1K; [11] used ImageNet-21k for testing few-shot learning; [48] tested efficient softmax schemes on ImageNet-21k; [15] teste”
BlackOut: Speeding up Recurrent Neural Network Language Models With Very Large Vocabularies (2015, methodology): “(2015), and the use of NCE (Gutmann & Hyvärinen, 2012) in Mnih & Teh (2012) all fall under this category, so does the more recent use of Locality Sensitive Hashing (LSH) techniques (Shrivastava & Li, 2014; Vijayanarasimhan et al., 2014) to select a subset of good samples.”
Compressing Gradient Optimizers via Count-Sketches (2019, methodology): “For example, (Shrivastava & Li, 2014; Vijayanarasimhan et al., 2014; Spring & Shrivastava, 2017; Yen et al., 2018a) have proposed using approximate nearest-neighbor search to find the most relevant output classes.”
One-vs-Each Approximation to Softmax for Scalable Estimation of Probabilities (2016, methodology): “We show that the new bound has interesting theoretical properties and we demonstratei s use in classification problems.”
Augment and Reduce: Stochastic Inference for Large Categorical Distributions (2018, methodology): “These include methods that approximate the exact computations (Gopal & Yang, 2013; Vijayanarasimhan et al., 2014), those that rely on sampling (Bengio & Sénécal, 2003; Mikolov et al.”
Beyond One-hot Encoding: lower dimensional target embedding (2018, background): “This constitutes an additional challenge for training CNN models in large output spaces, and the reason of slow convergence rates [40].”
Sampled Softmax with Random Fourier Features (2019, methodology): “One can also use an LSH algorithm to find the approximate nearest classes in the embedding space which in turn helps in sampling from the softmax distribution efficiently [11].”
Hierarchical Memory Networks (2016, methodology): “WTA-Hash: Winner Takes All hashing (Vijayanarasimhan et al., 2014) is a hashing-based K -MIPS algorithm which also converts MIPS to MCSS by augmenting additional dimensions to the vectors.”