Shark: SQL and rich analytics at scale

By Reynold Xin, Josh Rosen, Matei A. Zaharia, M. Franklin, Scott Shenker, Ion Stoica. ACM SIGMOD Conference, 2012

491 citations recorded by the source corpus.

Shark is a new data analysis system that marries query processing with complex analytics on large clusters. It leverages a novel distributed memory abstraction to provide a unified engine that can run SQL queries and sophisticated analytics functions (e.g. iterative machine learning) at scale, and efficiently recovers from failures mid-query. This allows Shark to run SQL queries up to 100X faster than Apache Hive, and machine learning programs more than 100X faster than Hadoop. Unlike previous systems, Shark shows that it is possible to achieve these speedups while retaining a MapReduce-like execution engine, and the fine-grained fault tolerance properties that such engine provides. It extends such an engine in several ways, including column-oriented in-memory storage and dynamic mid-query replanning, to effectively execute SQL. The result is a system that matches the speedups reported for MPP analytic databases over MapReduce, while offering fault tolerance properties and complex analytics capabilities that they lack.

Subfields: Computer Networks and Communications, Information Systems, Management Science and Operations Research

View Paper: https://www.semanticscholar.org/paper/f7c3b238c3041b1e6063875eaebdd8b2426cc73b

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Information Systems

In Management Science and Operations Research

Citations in the corpus

Big data analytics on Apache Spark (2016, background): “Spark SQL started from Shark project [84], and then, it became an essential library in Apache Spark.”
Efficient Coflow Scheduling Without Prior Knowledge (2015, methodology): “We thank Yuan Zhong, Ali Ghodsi, Shivaram Venkataraman, CCN members, our shepherd Hitesh Ballani, and the anonymous reviewers of NSDI’15 and SIGCOMM’15 for useful feedback, and Kay Ousterhout for generating Shark query plans for the TPC-DS queries.”
Fides: Towards a Platform for Responsible Data Science (2017, background): “In particular, recent advances in data systems research in new uses of existing technology [20], new architectures [24, 33], new execution strategies [8], and language extensions and interfaces [10, 32], drive the democratization of scal-”
Social big data: Recent achievements and new challenges (2015, background): “his framework is particularly well suited to machine learning algoithms [[29]].”
Sparrow: distributed, low latency scheduling (2013, methodology): “Each TPC-H query is executed using Shark [24], a large scale data analytics platform built on top of Spark [26].”
Knowledge Graphs (2020, unknown): “Amongst these we can mention Apache Spark (GraphX) [119, 562], GraphLab [325], Pregel [334], Signal–Collect [502], Shark [563], etc.”
Spark SQL: Relational Data Processing in Spark (2015, methodology): “With the experience from Shark, we wanted to extend relational processing to cover native RDDs in Spark and a much wider range of data sources.”