Interpreting the data: Parallel analysis with Sawzall

By Rob Pike, Sean M. Dorward, Robert Griesemer, Sean Quinlan. Scientific Programming, 2005

704 citations recorded by the source corpus.

Very large data sets often have a flat but regular structure and span multiple disks and machines. Examples include telephone call records, network logs, and web document repositories. These large data sets are not amenable to study using traditional database techniques, if only because they can be too large to fit in a single relational database. On the other hand, many of the analyses done on them can be expressed using simple, easily distributed computations: filtering, aggregation, extraction of statistics, and so on. We present a system for automating such analyses. A filtering phase, in which a query is expressed using a new procedural programming language, emits data to an aggregation phase. Both phases are distributed over hundreds or even thousands of computers. The results are then collated and saved to a file. The design -- including the separation into two phases, the form of the programming language, and the properties of the aggregators -- exploits the parallelism inherent in having data and computation distributed across many machines.

Subfields: Computer Networks and Communications

View Paper: https://www.semanticscholar.org/paper/ce56688ea6cf1761425d407ed4cf1c5d12e01212

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

Citations in the corpus

Dryad: distributed data-parallel programs from sequential building blocks (2007, neutral): “As a result, of the recent related systems it shares the most similarities with Google’s MapReduce [16, 33] which addresses a similar”
Google's MapReduce programming model - Revisited (2007, methodology): “Keywords: Data processing; Parallel programming; Distributed programming; Software design; Executable specification; Typed functional programming; MapReduce ; Sawzall; Map; Reduce; List homomorphism; Fold; Unfold; Bananas; Haskell.”
The Stratosphere platform for big data analytics (2014, background): “MapReduce has been the compilation target of SQL subsets [19,64], as well as other languages inspired by scripting languages [57,58] or XQuery [13].”
Tradeoffs between power management and tail latency in warehouse-scale applications (2014, background): “latency-insensitive saw String parsing in the Sawzall domain-specific language [24].”
Bigtable: A Distributed Storage System for Structured Data (2008, methodology): “The scripts are written in a language developed at Google for processing data called Sawzall [28].”
Toward Scalable Systems for Big Data Analytics: A Technology Tutorial (2014, background): “Several high-level language systems, such as Google’s Sawzall [168], Yahoo’s Pig Latin [169], Facebook’s Hive [170], and Microsoft’s Scope [132], have been proposed to improve programmersąŕ productivity.”
Improving MapReduce Performance in Heterogeneous Environments (2008, neutral): “Systems designed for SQL-like queries on top of MapReduce, such as Sawzall [9] and Pig [10], underline the importance of MapReduce for ad-hoc queries.”
Building a HighLevel Dataflow System on top of MapReduce: The Pig Experience (2009, neutral): “That need appears to be widespread, as several other systems are emerging that also oﬀer high-level languages for Map-Reduce-like environments: DryadLINQ [20], Hive [3], Jaql [5], Sawzall [16] and Scope [6].”
Large-scale distributed L-BFGS (2017, background): “Some high-level languages, such as Sawzall [26] and Yahoo Pig’s system [27], address some of the limitations of the MapReduce model by providing an external DataFloworiented programming language that is eventually translated into MapReduce processing sequences.”
DRAM errors in the wild: a large-scale field study (2009, methodology): “We solve this problem by using a parallel pre-processing step (implemented in Sawzall [18]), which runs on several hundred nodes simultaneously and performs basic data clean-up and filtering.”
Bigtable: a distributed storage system for structured data (2006, methodology): “The scripts are written in a language developed at Google for processing data called Sawzall [28].”
Pig latin: a not-so-foreign language for data processing (2008, background): “” Sawzall [13] is a scripting language used at Google on top of map-reduce.”
A Survey of Large Scale Data Management Approaches in Cloud Environments (2011, neutral): “2) Sawzall: Sawzall [68] is a scripting language used at Google on top of MapReduce.”
Boa: A language and infrastructure for analyzing ultra-large-scale software repositories (2013, background): “Sizzle is an opensource Java implementation of the Sawzall language.”