Interactive Analytical Processing in Big Data Systems: A Cross-Industry Study of MapReduce Workloads

By Yanpei Chen, Sara Alspaugh, Randy Howard Katz. Proceedings of the VLDB Endowment, 2012

570 citations recorded by the source corpus.

Within the past few years, organizations in diverse industries have adopted MapReduce-based systems for large-scale data processing. Along with these new users, important new workloads have emerged which feature many small, short, and increasingly interactive jobs in addition to the large, long-running batch jobs for which MapReduce was originally designed. As interactive, large-scale query processing is a strength of the RDBMS community, it is important that lessons from that field be carried over and applied where possible in this new domain. However, these new workloads have not yet been described in the literature. We fill this gap with an empirical analysis of MapReduce traces from six separate business-critical deployments inside Facebook and at Cloudera customers in e-commerce, telecommunications, media, and retail. Our key contribution is a characterization of new MapReduce workloads which are driven in part by interactive analysis, and which make heavy use of query-like programming frameworks on top of MapReduce. These workloads display diverse behaviors which invalidate prior assumptions about MapReduce such as uniform data access, regular diurnal patterns, and prevalence of large jobs. A secondary contribution is a first step towards creating a TPC-like data processing benchmark for MapReduce.

Subfields: Computer Networks and Communications, Computer Vision and Pattern Recognition, Information Systems

View Paper: https://www.semanticscholar.org/paper/fb5323ebb302a9ebc14d4b292b55087923c52e95

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Computer Vision and Pattern Recognition

In Information Systems

Citations in the corpus

The rise of "big data" on cloud computing: Review and open research issues (2015, background): “MapReduce-based systems have been adopted by many organizations in the past few years for long-running batch jobs [27].”
Tachyon: Reliable, Memory Speed Storage for Cluster Computing Frameworks (2014, methodology): “While some jobs’ outputs are much smaller than their inputs, a recent trace from Cloudera showed that, on average, 34% of jobs (weighted by compute time) across ﬁve customers had outputs that were at least as large as their inputs [20].”
Tupleware: Distributed Machine Learning on Small Clusters (2014, background): “For instance, the average Cloudera customer rarely works with datasets larger than a few terabytes in size [15], and commonly analyzed behavioral data peaks at around 1TB [11].”
Big Data computing and clouds: Trends and future directions (2013, background): “Previous work shows that there is an emerging class of MapReduce applications that feature small, short, and highly interactive jobs [73, 74].”
Toward Scalable Systems for Big Data Analytics: A Technology Tutorial (2014, background): “More complex analysis for production workload traces can be found in the authors’ subsequent research [289].”