Twister: a runtime for iterative MapReduce

By Jaliya Ekanayake, Bingjing Zhang, Thilina Gunarathne, Seung-Hee Bae, Judy Qiu, Geoffrey C. Fox. IEEE International Symposium on High-Performance Parallel Distributed Computing, 2010

1,021 citations recorded by the source corpus.

MapReduce programming model has simplified the implementation of many data parallel applications. The simplicity of the programming model and the quality of services provided by many implementations of MapReduce attract a lot of enthusiasm among distributed computing communities. From the years of experience in applying MapReduce to various scientific applications we identified a set of extensions to the programming model and improvements to its architecture that will expand the applicability of MapReduce to more classes of applications. In this paper, we present the programming model and the architecture of Twister an enhanced MapReduce runtime that supports iterative MapReduce computations efficiently. We also show performance comparisons of Twister with other similar runtimes such as Hadoop and DryadLINQ for large scale data parallel applications.

Subfields: Computer Networks and Communications, Computer Vision and Pattern Recognition, Hardware and Architecture

View Paper: https://www.semanticscholar.org/paper/09a180d9d410d8e551f42401d6453d57406b6d29

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Computer Vision and Pattern Recognition

In Hardware and Architecture

Citations in the corpus

Dandelion: a compiler and runtime for heterogeneous systems (2013, background): “Extending dataflow systems to support iteration [21, 40, 70, 75] or incremental iterative computation [72, 41, 73, 74] is an active research area.”
Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center (2011, background): “Prominent examples include MapReduce [18], Dryad [24], MapReduce Online [17] (which supports streaming jobs), Pregel [28] (a specialized framework for graph computations), and others [27, 19, 30].”
Big data analytics on Apache Spark (2016, neutral): “, HaLoop [13] and Twister [26]), GraphLab [56] PowerGraph [34], Google’s Pregel [58] (a Pregel API was implemented on top of Spark core [94]) and of course Spark’s MLlib [61].”
PowerGraph: Distributed Graph-Parallel Computation on Natural Graphs (2012, neutral): “5B 50x2 Twister [15] 36s 50M 1.”
MapReduce for Parallel Reinforcement Learning (2011, methodology): “We also observe that the current Hadoop and its iterative proposals, including Haloop and Twister, are not general enough to support certain iteration styles in a natrual way, for example, for LSPI and TD(λ).”
WTF: the who to follow service at Twitter (2013, neutral): “Most of these issues can be addressed in a more principled way by extending the MapReduce programming model, for example, HaLoop [3], Twister [7], and PrIter [31].”
The rise of "big data" on cloud computing: Review and open research issues (2015, methodology): “Twister [76] provides support for efficient and iterative MapReduce computations.”
Coded MapReduce (2015, methodology): “1) Server 3 sends intermediate pairs (A, 11)[7], (A, 14)[8], (A, 10)[9], (A, 15)[10], (A, 15)[11], (A, 10)[12] and then (B, 21)[7], (B, 21)[8], (B, 21)[9], (B, 21)[10], (B, 21)[11], (B, 21)[12].”
GraphJet: Real-Time Content Recommendations at Twitter (2016, background): “There were a few academic research papers on iterative formulations of MapReduce [12, 41], but nothing that was close to production ready.”
Big Data Processing in Cloud Computing Environments (2012, neutral): “Twister[33] proposed by J.”
Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing (2012, methodology): “The systems keep data partitioned consistently across iterations, and Twister can also keep it in memory.”
One Trillion Edges: Graph Processing at Facebook-Scale (2015, background): “Iterative frameworks on MapReduce style computing models is an area that has been explored in Twister [18] and Haloop [13].”
Privacy-Preserving Machine Learning Algorithms for Big Data Systems (2015, methodology): “Notice that Hadoop MapReduce may not be suitable for iterative computation tasks, however, in [12], Ekanayake and Li propose an iterative version of MapReduce called Twister, which could be used to achieve iterative update in (13).”
Toward Scalable Systems for Big Data Analytics: A Technology Tutorial (2014, background): “In addition, other research has focused on programming models for more specific tasks, such as cross joining two sets [173], iterative computation [174], [175], in-memory computation with fault-tolerance [176], incremental computation [177]–[180], and data-dependent flow control decision [181].”
Tupleware: Distributed Machine Learning on Small Clusters (2014, background): “In contrast, a number of iterative extensions to MapReduce have been proposed (e.g., Stratosphere, HaLoop [10], Twister [18]), but these approaches either lack low-level optimization potential or do not scale well.”