Heterogeneity-aware Distributed Parameter Servers

By Jiawei Jiang, Bin Cui, Ce Zhang, Lele Yu. 2017

219 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Information Systems

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Information Systems

Citations in the corpus

Demystifying Parallel and Distributed Deep Learning (2018, methodology): “Parameter servers also enable handling heterogeneity, both in training agents [125] and in network settings (e.”
HetPipe: Enabling Large DNN Training on (Whimpy) Heterogeneous GPU Clusters through Integration of Pipelined Model Parallelism and Data Parallelism (2020, methodology): “As it is important that the results generated through our proposed system configuration are correct [16, 21, 54], we show the convergence of our methodology in Section 6.”
Online Machine Learning in Big Data Streams (2018, background): “Angel supports synchronous, asynchronous, and stale synchronous processing [125].”
Efficient Memory Management for GPU-based Deep Learning Systems (2019, methodology): “In the future work, we will extend our solutions for the applications whose iterations have slight variations, explore memory reduction in a distributed environment [44], and adapt our approaches into other memory-hungry applications with iterative nature, such as large-scale K-Means running on GPU.”
Asynchronous Byzantine Machine Learning (2018, methodology): “Additionally, we compare with a state-of-the-art staleness-aware learning algorithm (D YN SGD (Jiang et al., 2017)) that employs an inverse dampening function ( Λ 2 ( τ ) = + τ ).”
Hop: Heterogeneity-aware Decentralized Training (2019, background): “In general, it involves random aspects such as slowdown caused by resource sharing and hardware faults, as well as deterministic factors including differences in hardware computational capabilities and network bandwidths [15, 16, 21].”
SparCML: High-Performance Sparse Communication for Machine Learning (2018, unknown): “Note that the value of τ can adversely impact the convergence rate of various algorithms [26, 34, 40].”