Distributed GraphLab: A Framework for Machine Learning in the Cloud

By Yucheng Low, Joseph E. Gonzalez, Aapo Kyrola, Danny Bickson, Carlos Guestrin, Joseph M Hellerstein. 2012

1,087 citations recorded by the source corpus.

While high-level data parallel frameworks, like MapReduce, simplify the design and implementation of large-scale data processing systems, they do not naturally or efficiently support many important data mining and machine learning algorithms and can lead to inefficient learning systems. To help fill this critical void, we introduced the GraphLab abstraction which naturally expresses asynchronous, dynamic, graph-parallel computation while ensuring data consistency and achieving a high degree of parallel performance in the shared-memory setting. In this paper, we extend the GraphLab framework to the substantially more challenging distributed setting while preserving strong data consistency guarantees. We develop graph based extensions to pipelined locking and data versioning to reduce network congestion and mitigate the effect of network latency. We also introduce fault tolerance to the GraphLab abstraction using the classic Chandy-Lamport snapshot algorithm and demonstrate how it can be easily implemented by exploiting the GraphLab abstraction itself. Finally, we evaluate our distributed implementation of the GraphLab abstraction on a large Amazon EC2 deployment and show 1-2 orders of magnitude performance gains over Hadoop-based implementations.

Subfields: Computer Vision and Pattern Recognition, Hardware and Architecture, Information Systems, ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Hardware and Architecture

In Information Systems

In ML Systems & Hardware

Citations in the corpus

Towards Federated Learning at Scale: System Design (2019, background): “Many systems support multiple distribution schemes, including model parallelism and data parallelism , e.g., Dean et al. (2012) and Low et al. (2012).”
Distributed Statistical Machine Learning in Adversarial Settings (2017, background): “Many efficient distributed machine learning algorithms [6, 13] and system implementations [24, 28, 10, 19] have been proposed and studied.”
AIDE: Fast and Communication Efficient Distributed Optimization (2016, methodology): “For example, large scale distributed machine learning systems such as the Parameter server [9], GraphLab [22] and TensorFlow [1] work with datasets sizes in the order of hundreds of terabytes.”
CADD: predicting the deleteriousness of variants throughout the human genome (2018, positive): “0 used a linear support vector machine implemented in the LIBOCAS library (42), later models used L 2 -regularized logistic regression implemented in GraphLab Create (43).”
Quantitative missense variant effect prediction using large-scale mutagenesis data (2017, positive): “…and Algorithms Python/2.7.3 Python https://www.python.org/ Numpy/1.13.1 van der Walt et al. (2011) http://www.numpy.org/ GraphLab/2.1 Low et al. (2012) https://turi.com/ Scipy/0.19.1 Jones et al. (2001) https://www.scipy.org/ scikit-learn/0.17.0 Pedregosa et al. (2011)…”
More Effective Distributed ML via a Stale Synchronous Parallel Parameter Server (2013, background): “(2) Building high-throughput distributed ML architectures or algorithm implementations that feature significant systems contributions but relatively less theoretical analysis, such as GraphLab [18], Spark [27], Pregel [19], and YahooLDA [2].”
A survey of machine learning for big data processing (2016, methodology): “In this context, distributed GraphLab, a framework for machine learning in the cloud, has been proposed in [80].”
Reliable Data Distillation on Graph Convolutional Network (2020, background): “Graph data has been widely studied in recent years [28, 38, 40], and a large number of graph-based semi-supervised learning methods have been proposed since labeling an entire graph is extremely time-consuming.”
Addressing the straggler problem for iterative convergent parallel ML (2016, positive): “For example, GraphLab [38, 39] programs structure computation as”
Communication Efficient Distributed Machine Learning with the Parameter Server (2014, methodology): “Distributed optimization is becoming a key tool for solving large scale machine learning problems [1] [3] [10] [21] [19].”
Shark: SQL and rich analytics at scale (2012, methodology): “First, many complex analytics functions, such as machine learning and graph algorithms, are iterative, going over the data multiple times; thus, the fastest systems deployed for these applications are in-memory [23, 22, 33].”
Fault Tolerance in Iterative-Convergent Machine Learning (2018, background): “One notable exception to this trend is checkpoint-based fault tolerance, a common strategy in current ML systems for mitigating hardware failures (Abadi et al., 2016; Wei et al., 2015; Low et al., 2012) which continues to enforce strong consistency semantics at a high cost of re-computing lost work.”
On Convergence of Model Parallel Proximal Gradient Algorithm for Stale Synchronous Parallel System (2016, background): “Existing systems can roughly be divided into three categories: bulk synchronous [14, 34, 37], (totally) asynchronous [8, 25], and partially asynchronous (also called stale synchronous in this work) [2, 8, 16, 19, 21, 32].”