Improving MapReduce Performance in Heterogeneous Environments

By Matei A. Zaharia, Andy Konwinski, Anthony D. Joseph, Randy Howard Katz, Ion Stoica. 2008

1,941 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

DRACO: Byzantine-resilient Distributed Training via Redundant Gradients (2018, background): “Synchronous training can suffer from straggler nodes (Zaharia et al., 2008), where a few compute nodes are significantly slower than average.”
Speeding Up Distributed Machine Learning Using Codes (2015, methodology): “[27] propose a modification to the existing straggler detection algorithm and show that the proposed solution can effectively reduce the completion time of MapReduce tasks.”
Polynomial Codes: an Optimal Design for High-Dimensional Coded Matrix Multiplication (2017, background): “For example, replicating the straggling task on another available node is a common approach to deal with stragglers (e.g., [4]).”
Incoop: MapReduce for incremental computations (2011, background): “, individual poorly performing nodes that can drastically delay the overall job completion [22].”
CPI2: CPU performance isolation for shared compute clusters (2013, background): “Although identifying laggards and starting up replacements for them in a timely fashion [39, 3] often improves performance, it typically does so at the cost of additional resources.”
Solving the Straggler Problem with Bounded Staleness (2013, background): “For consistent stragglers (e.g., less capable nodes), proper load distribution (e.g., via work stealing) or speculative execution (e.g., [22]) sufﬁces.”
Cloud computing and emerging IT platforms: Vision, hype, and reality for delivering computing as the 5th utility (2009, background): “[45] shows that MapReduce implemented in the standard distribution of Hadoop is unable to performwell in heterogeneous Cloud computing infrastructure such as Amazon EC2 [29].”
Making Sense of Performance in Data Analytics Frameworks (2015, methodology): “This has driven work on mitigation using task speculation [8] [10] [11] [52] or running shorter tasks to improve load balancing [39].”
Using Straggler Replication to Reduce Latency in Large-scale Parallel Computing (2015, background): “A line of systems work [2, 3, 15, 23] further developed this idea to handle various performance variability issues in data centers.”
Coded computation over heterogeneous clusters (2017, background): “, [3]), while partial data replication is also used to reduce the communication load in distributed computing (e.”
Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center (2011, background): “Second, as new frameworks and new scheduling policies for current frameworks are constantly being developed [37, 38, 40, 26], it is not clear whether we are even at the point to have a full specification of framework requirements.”
Performance Isolation and Fairness for Multi-Tenant Cloud Storage (2012, unknown): “, in response time or throughput, even when they can achieve their desired mean rate [8, 16, 33, 35].”
Low Latency Geo-distributed Data Analytics (2015, background): “These systems improve data locality of input tasks and fairness [37, 58], and minimize outliers in task execution [16, 19, 61].”
Sharing the Data Center Network (2011, background): “This is because schedulers for map-reduce platforms [27, 45] explicitly allocate local resources such as compute slots and memory.”
Gradient Coding: Avoiding Stragglers in Distributed Learning (2017, background): “There do exist general systems solutions such as speculative execution (Zaharia et al., 2008) but we believe that approaches tailored to machine learning can be vastly more efficient.”
Resource Management in Clouds: Survey and Research Challenges (2015, background): “[260] showed that Hadoop’s default task scheduler implementation performed badly when deployed on a public IaaS cloud due to its implicit assumption that compute resources available to it are homogeneous and that tasks make progress linearly.”
Reining in the Outliers in Map-Reduce Clusters using Mantri (2010, methodology): “This cluster is up to two orders of magnitude larger than those in previous publications [1, 12, 20] and exhibits a high level of concurrency due to many jobs simultaneously running on the cluster and many tasks on a machine.”