Paleo: A Performance Model for Deep Neural Networks

By Hang Qi, Evan R. Sparks, Ameet Talwalkar. 2016

252 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

Beyond human-level accuracy: computational challenges in deep learning (2019, methodology): “oarse approximations of FLOPs for a range of applications ranging from AlexNet to Alpha Go [1]. PerformanceModelingPaleo is an analytical performance model which explores parallelism for CNN networks [27]. 8Conclusion This paper leverages the prior work to project the dataset and model size growth r”
Federated Multi-Task Learning (2017, methodology): “Detailed models of this type have been used to closely match the performance of real-world systems [38].”
ATOMO: Communication-efficient Learning via Atomic Sparsification (2018, background): “It is now widely acknowledged that communication overheads are the main source of this speedup saturation phenomenon [4, 5, 6, 7, 8].”
Demystifying Parallel and Distributed Deep Learning (2018, methodology): “ CPUs in a distributed environment[247], using a similar approach, and for Intel Xeon Phi accelerators[235] strictly for training time estimation (i.e., not individual layers or DNN evaluation). Paleo[196] derives a performance model from operation counts alone (with 10–30% prediction error), and Pe”
A System for Massively Parallel Hyperparameter Tuning (2018, methodology): “Qi et al. (2017) speedups for a variety of models.”
The Effect of Network Width on the Performance of Large-batch Training (2018, background): “In practice, these gains usually saturate beyond a few tens of compute nodes [6, 10, 11].”
Buddy Compression: Enabling Larger Memory for Deep Learning and HPC Workloads on GPUs (2019, methodology): “This simulator, similar to Paleo [57], uses analytical models to project DL processing throughput as workload and architectural parameters are varied.”
ILFO: Adversarial Attack on Adaptive Neural Networks (2020, methodology): “2For total FLOPs calculation, we calculate FLOPs for each convolutional layer [24]”
Approximate Gradient Coding via Sparse Random Graphs (2017, background): “Unfortunately, these speedup gains often fall short of the theoretically optimal speedup when scaling beyond a few tens of compute nodes [5, 19].”
Parle: parallelizing stochastic gradient descent (2017, background): “Both synchronous and asynchronous gradient aggregation necessitate a high per-worker load (Qi et al., 2016) if they are to scale well.”
NeuralPower: Predict and Deploy Energy-Efficient Convolutional Neural Networks (2017, methodology): “A work that shares similar insight with our methodology is the Paleo framework proposed by Qi et al. (2016).”
Gradient Diversity Empowers Distributed Learning (2017, background): “Several studies [9, 35] indicate that there is a significant gap between ideal and realizable speedups when scaling out to hundreds of compute nodes.”
Predicting the Computational Cost of Deep Learning Models (2018, background): “For example the PALEO [3] system computes the number of floating point operations required for an epoch and multiplies this by a scaling factor derived from testing the floating point operation speed on a given system.”
HetPipe: Enabling Large DNN Training on (Whimpy) Heterogeneous GPU Clusters through Integration of Pipelined Model Parallelism and Data Parallelism (2020, methodology): “For the given data size, we predict intra-node communication based on the PCI-e bandwidth, then multiply it by a scaling-down constant (which is similarly done in Paleo [41]), since in practice, it is not possible to utilize the peak bandwidth.”
Gradient Diversity: a Key Ingredient for Scalable Distributed Learning (2017, background): “Several studies (Dean et al., 2012, Qi et al., 2016) indicate that there is a signiﬁ-cant gap between ideal and realizable speedups when scaling out to hundreds of compute nodes.”
Gradient Coding Using the Stochastic Block Model (2018, background): “While distributed algorithms can theoretically achieve substantial speedups, in practice the parallelization gains often fall short of the optimal speedup gains predicted by theory [1], [2].”