Gaia: Geo-Distributed Machine Learning Approaching LAN Speeds

By Kevin Hsieh, Aaron Harlap, Nandita Vijaykumar, Dimitris Konomis, G. R. Ganger, Phillip B. Gibbons, Onur Mutlu. 2017

450 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In ML Systems & Hardware

Citations in the corpus

The Non-IID Data Quagmire of Decentralized Machine Learning (2019, methodology): “…traditional training approaches (i.e., those designed for centralized data) requires massive communication, which drastically slows down the training process because the communication is bottlenecked by the limited wide-area or mobile network bandwidth (Hsieh et al., 2017; McMahan et al., 2017).”
Learning Privately over Distributed Features: An ADMM Sharing Approach (2019, background): “Recently, a wide range of distributed and collaborative machine learning schemes, including gradient-based meth-ods Li et al. (2014a,b); Hsieh et al. (2017); Ho et al. (2013) and ADMM-based methods Zhang et al. (2018); Shi et al. (2014); Zhang and Zhu (2016a); Huang et al. (2018), have been…”
Adaptive Federated Learning in Resource Constrained Edge Computing Systems (2018, background): “Although the work in [24] is related to the adaptation of synchronization frequency with resource considerations, it focuses on peer-to-peer connected datacenters, which is different from the federated learning architecture that is not peer-to-peer.”
Demystifying Parallel and Distributed Deep Learning (2018, background): “The latter work [109] acknowledges that training may be geo-distributed, i.”
FDML: A Collaborative Machine Learning Framework for Distributed Features (2019, background): “A similar bounded delay condition is enforced in most Parameter-Server-like systems [1, 5, 14, 17, 19] to ensure convergence and avoid chaotic behavior of a completely asynchronous system.”
A Survey on Distributed Machine Learning (2019, background): “A disadvantage is that it can be hard to choose the parameter that denes which update are signicant and which are not [73].”
Biscotti: A Blockchain System for Private and Secure Federated Learning (2021, background): “However, when training ML models in a multi-party setting, users must share their potentially sensitive information with a centralized service [1], [2], [3], [4].”
AGGREGATHOR: Byzantine Machine Learning via Robust Gradient Aggregation (2019, background): “This order of magnitude for the drop ratio, although high for a data-center environment, can be realistic in a WAN environment (Kumar et al., 2015) for distributed machine learning (Hsieh et al., 2017).”
The Limitations of Federated Learning in Sybil Settings (2020, background): “In a data inversion attack, the adversary reconstructs the training data of a targeted honest client [27] by generating a sample that closely represents the training data of a specific client.”
A Survey on the Convergence of Edge Computing and AI for UAVs: Opportunities and Challenges (2022, methodology): “In such scenarios, the aggregation process can be carefully controlled and optimized through “aggregation frequency control” [8], [40].”
Distributed Machine Learning in Coalition Environments: Overview of Techniques (2018, methodology): “For example, [11] proposed a method that sends the model weights to other nodes only when the change in the local model weight exceeds a threshold.”
Heterogeneity-Aware Asynchronous Decentralized Training (2019, background): “The trend of heterogeneity and the “long tail effects” have been also discussed and confirmed in other recent works [5,12,24,28,35].”
Communication-Efficient Decentralized Learning with Sparsification and Adaptive Peer Selection (2020, background): “Gaia [7] is another form of gradient compression, who filters out “insignificant” updates.”
Unleashing the Power of Edge-Cloud Generative AI in Mobile Networks: A Survey of AIGC Services (2023, unknown): “[135] Geo-distributed ML Execution time/Selective barrier, mirror clock Provisioning Localized AIGC services Convergence analysis”
Hop: Heterogeneity-aware Decentralized Training (2019, background): “However, it falls short on support for heterogeneity, although various approaches have been proposed in centralized training to deal with this problem [3, 14, 15, 16, 23].”
Demo abstract: Distributed machine learning at resource-limited edge nodes (2018, background): “A distributed learning system that adapts the global aggregation frequency is presented in [1], where aggregation is performed whenever the model weight varies beyond an empirically selected threshold.”
Distributed Hierarchical GPU Parameter Server for Massive Scale Deep Learning Ads Systems (2020, background): “Posei-don (Zhang et al., 2017) develops two strategies wait-free back-propagation and hybrid communication to exploit the independency between parameters of layered model structures in deep neural networks and conquer communication bottleneck; while Gaia (Hsieh et al., 2017) even considered the prob”
Deep Learning With Edge Computing: A Review (2019, background): “Gaia [127] studies synchronous SGD in the scenario where devices are geographically distributed across a large area.”