Multi-GPU Training of ConvNets

By Omry Yadan, Keith Adams, Yaniv Taigman, Marc'Aurelio Ranzato. 2013

105 citations recorded by the source corpus.

In this work, we consider a standard architecture [1] trained on the Imagenet dataset [2] for classification and investigate methods to speed convergence by parallelizing training across multiple GPUs. In this work, we used up to 4 NVIDIA TITAN GPUs with 6GB of RAM. While our experiments are performed on a single server, our GPUs have disjoint memory spaces, and just as in the distributed setting, communication overheads are an important consideration. Unlike previous work [9, 10, 11], we do not aim to improve the underlying optimization algorithm. Instead, we isolate the impact of parallelism, while using standard supervised back-propagation and synchronous mini-batch stochastic gradient descent.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

One weird trick for parallelizing convolutional neural networks (2014, methodology): “In [Yadan et al., 2013], the authors parallelize the training of the convolutional neural net from [Krizhevsky et al.”
Deep learning with Elastic Averaging SGD (2014, background): “This paper focuses on the problem of reducing the parameter communication overhead between the master and local workers [10, 2, 11, 12, 13].”
Deep Image: Scaling up Image Recognition (2015, methodology): “Similar strategies have been proposed in the previous work (Krizhevsky, 2014), (Yadan et al., 2013).”
FireCaffe: Near-Linear Acceleration of Deep Neural Network Training on Compute Clusters (2015, neutral): “sized DNNs like AlexNet and NiN. Twitter [14] has also experimented with scaling DNN training to 8 GPUs, but speed and accuracy results have not been released. Tencent [49], Theano [12], and Facebook [47] have published single-server/multiGPU speed numbers that are slower than Google [26]. FireCaffe”
Demystifying Parallel and Distributed Deep Learning (2018, methodology): “ently, and thus the overall algorithm has to be Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis 1:23 Category Method Model Consistency Synchronization Synchronous[45,112,173,210,224,246] Stale-Synchronous[88,98,120,156,261] Asynchronous[55,56,127,182,190,205,260”
Theano-based Large-Scale Visual Recognition with Multiple GPUs (2014, background): “Many multi-GPU frameworks has been proposed and implemented (Yadan et al., 2013; Zou et al., 2014; Paine et al., 2013; Krizhevsky, 2014), usually adopting a mixed data and model parallelism.”
Deep Convolutional Neural Networks for Image Classification: A Comprehensive Review (2017, methodology): “Yadan et al. (2014) utilized a hybrid parallelism strategy that considered both model and data parallelism.”
Neural Networks and Deep Learning (2018, unknown): “Various types of parallelization tricks for GPUs are discussed in [74, 91, 254], and specific discussions on convolutional neural networks are provided in [541].”
Communication-Efficient and Distributed Learning Over Wireless Networks: Principles and Applications (2020, background): “In essence, this problem is traced back to model parallelism focusing on how to partition and offload NN segments, as opposed to data parallelism considering a large-sized global data set dispersed across different workers running NN models, each of which is separate but has the same architecture [1”
A Survey on Deep Learning (2018, background): “The two kinds of strategies can also be fused to a hybrid model as discussed in [168].”
Exploiting Feature and Class Relationships in Video Categorization with Regularized Deep Neural Networks (2015, background): “Several recent studies investigated this issue by proposing efficient classification methods [19], [20] or parallel computing strategies [21], [22].”
Federated Learning on Non-IID Data: A Survey (2021, background): “Although, data parallelism [113, 78, 11, 147, 42, 115] in distributed machine learning [23, 94, 64], which has a very similar learning scheme with horizontal FL, has been empirically proved to be valid and widely used in multi-GPU computation to accelerate the learning speed in deep learning, design”
Distributed stochastic optimization for deep learning (thesis) (2016, unknown): “We will focus on the problem of reducing the parameter communication overhead between the master and the local workers [50, 16, 60, 43, 48].”
Caffeine: Towards uniformed representation and acceleration for deep convolutional neural networks (2016, unknown): “2967011 [5], Facebook [6], and Baidu [7].”
Decoupled Parallel Backpropagation with Convergence Guarantee (2018, background): “Relation to model parallelism: Model parallelism usually refers to ﬁlter-wise parallelism (Yadan et al., 2013).”