Building a HighLevel Dataflow System on top of MapReduce: The Pig Experience

By Alan Gates, Olga Natkovich, Shubham Chopra, Pradeep Kamath, Shravan Narayanam, Christopher Olston, Benjamin C. Reed, Santhosh Srinivasan, Utkarsh Srivastava. Proceedings of the VLDB Endowment, 2009

342 citations recorded by the source corpus.

Subfields: Computer Networks and Communications, Information Systems, Information Systems and Management

View Paper: https://www.semanticscholar.org/paper/fe1cc4e034ad4d3a54a5aa0a53a24b6f564298c4

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Information Systems

In Information Systems and Management

Jimmy J. Lin

Citations in the corpus

Efficient parallel set-similarity joins using MapReduce (2010, methodology): “(A similar pattern was observed in [10] for the case of general aggregations.”
Large-scale distributed L-BFGS (2017, background): “Some high-level languages, such as Sawzall [26] and Yahoo Pig’s system [27], address some of the limitations of the MapReduce model by providing an external DataFloworiented programming language that is eventually translated into MapReduce processing sequences.”
SystemML: Declarative machine learning on MapReduce (2011, background): “Overall, CPMM performs very stably with increasing sizes of W’ and W. Piggybacking: To analyze the impact of piggybacking several lops into a single MapReduce job, we compare piggybacking to a naive approach, where each lop is evaluated in a separate MapReduce job.”
RCFile: A fast and space-efficient data placement structure in MapReduce-based warehouse systems (2011, methodology): “In Zebra used for the Pig System, a column group is actually row-oriented to reduce the overhead of a record reconstruction in a MapReduce environment.”
WTF: the who to follow service at Twitter (2013, methodology): “The Real Graph itself is constructed on a periodic basis through a series of Pig scripts that run on Hadoop, joining together data from disparate sources.”
The Hadoop Distributed File System (2010, background): “Pig [ 4 ], ZooKeeper [6], and Chukwa were originated and developed at Yahoo! Avro was originated at Yahoo! and is being co-developed with Cloudera.”
Hadoop++ (2010, methodology): “see results for the selection task in [12]).”
A Survey of Large Scale Data Management Approaches in Cloud Environments (2011, background): “[66] have presented a language called Pig Latin that takes a middle position between expressing task using high-level declarative querying model in the spirit of SQL and low-level/procedural programming using MapReduce.”
Interactive Analytical Processing in Big Data Systems: A Cross-Industry Study of MapReduce Workloads (2012, background): “Several recent studies offered single use case counter-points to the above mental model [10, 14, 17, 18, 27, 49].”
Hadoop-GIS: A High Performance Spatial Data Warehousing System over MapReduce (2013, background): “Declarative query interfaces such as Hive [32], Pig [21], and Scope [19] have brought the large scale data analysis one step closer to the common users by providing high level, easy to use programming abstractions to MapReduce.”