Large-scale deep unsupervised learning using graphics processors

By Rajat Raina, Anand Madhavan, Andrew Y. Ng. 2009

784 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics, Large Language Models, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistical and Nonlinear Physics

In Large Language Models

In Optimization & Training

Citations in the corpus

GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding (2020, background): “Since late 2000s, researchers started to leverage GPUs to accelerate neural nets [70, 57, 71].”
Deep Speech 2 : End-to-End Speech Recognition in English and Mandarin (2015, background): “Training on a single GPU resulted in substantial performance gains (Raina et al., 2009), which were subsequently scaled linearly to two (Krizhevsky et al.”
Building high-level features using large scale unsupervised learning (2011, methodology): “1 Our input data are 200x200 images, much larger than typical 32x32 images used in deep learning and unsupervised feature learning (Krizhevsky, 2009; Ciresan et al., 2010; Le et al., 2010; Coates et al., 2011).”
Deep Speech: Scaling up end-to-end speech recognition (2014, background): “Early ports of DL algorithms to GPUs revealed signiﬁcant speed gains [33].”
BinaryConnect: Training Deep Neural Networks with binary weights during propagations (2015, background): “Interestingly, one of the key factors that enabled this major progress has been the advent of Graphics Processing Units (GPUs), with speed-ups on the order of 10 to 30-fold, starting with [8], and similar improvements with distributed training [9, 10].”
Deep learning in neural networks: An overview (2014, methodology): “2009 also saw an impressive GPU-DBN implementation (Raina et al., 2009) orders of magnitudes faster than previous CPU-DBNs (see Sec.”
A Deep Learning Approach for Optimization of Systematic Signal Detection in Financial Trading Systems with Big Data (2017, methodology): “With recent advances in hardware and data storage, Big data processing technologies have made it more efficient and affordable to use complex machine learning algorithms in various computational problems [22], [23].”
Sparse Communication for Distributed Gradient Descent (2017, background): “Distributed computing is essential to train large neural networks on large data sets (Raina et al., 2009).”
Exponentially Increasing the Capacity-to-Computation Ratio for Conditional Computation in Deep Learning (2014, background): “An important factor in these recent breakthroughs has been the availability of GPUs which have allowed training deep nets at least 10 times faster, often more (Raina et al. , 2009).”
Deep Learning of Representations: Looking Forward (2013, background): “They have also taken advantage of GPU technology to speed-up computation by one or two orders of magnitude (Raina et al., 2009; Bergstra et al., 2010, 2011; Krizhevsky et al., 2012).”
On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models (2015, methodology): “Today’s faster computers, such as GPUs, mitigate the Fundamental Deep Learning Problem for FNNs [181, 34, 198, 38, 40].”
Demystifying Parallel and Distributed Deep Learning (2018, methodology): “One of the earliest occurrences of mapping data parallel DNN computations to GPUs were performed in [Raina et al. 2009].”
Next generation cloud computing: New trends and research directions (2017, background): “The availability of hardware accelerators, such as GPUs, in cloud environments has reduced the computing time for machine learning algorithms on large volumes of data [81, 82, 83].”
State-of-the-Art Deep Learning: Evolving Machine Intelligence Toward Tomorrow’s Intelligent Network Traffic Control Systems (2017, background): “…computing architectures exploiting GPUs for efﬁciently constructing incredibly large-scale deep learning models containing millions of connection weights [247], [248] capable of being trained in an unsupervised manner by using the huge number of patterns in the currently available datasets [33].”
Deep Neural Networks Motivated by Partial Differential Equations (2018, background): “By design, each CNN layer exploits the local relation between image information, which simpliﬁes computation [39].”
A Survey of the Usages of Deep Learning for Natural Language Processing (2020, background): “Recent increases in computational power and parallelization, harnessed by graphical processing units (GPUs) [3], [4], now allow for “deep learning,” which utilizes artificial neural networks (ANNs), sometimes with billions of trainable parameters [5].”