Spark SQL: Relational Data Processing in Spark

By Michael Armbrust, Reynold Xin, Cheng Lian, Yin Huai, Davies Liu, Joseph K. Bradley, Xiangrui Meng, Tomer Kaftan, M. Franklin, A. Ghodsi and 1 more. SIGMOD Conference, 2015

1,484 citations recorded by the source corpus.

Subfields: Computer Networks and Communications, Information Systems, Information Systems and Management

View Paper: https://www.semanticscholar.org/paper/ada0b87cd5c30d31186c38fb12e631d29426a3bf

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Information Systems

In Information Systems and Management

Citations in the corpus

Sonata: query-driven streaming network telemetry (2018, background): “Database research has explored query planning and optimization extensively [4, 32, 36].”
BigBench V2: The New and Improved BigBench (2017, methodology): “Unlike SparkSQL, Drill does not support HiveQL and we implemented the queries using the Drill’s native JSON support (Section V).”
Learning to Optimize Join Queries With Deep Reinforcement Learning (2018, methodology): “We implement three versions of DQ to illustrate the ease of integration into existing DBMSes: (1) A standalone version built on top of Apache Calcite [2], (2) a version integrated with PostgreSQL [3], and (3) a version integrated with SparkSQL [7].”
A Survey of Traffic Prediction: from Spatio-Temporal Data to Intelligent Transportation (2021, methodology): “Differently, GeoSpark does not support Spark SQL [86] or the DataFrame API, while Simba can support.”
SparkNet: Training Deep Networks in Spark (2015, background): “…SparkNet, training a deep network on the output of a SQL query, or a graph computation, or a streaming data source is straightforward due to its general purpose nature and its support for SQL, graph computations, and data streams (Armbrust et al., 2015; Gonzalez et al., 2014; Zaharia et al., 2013).”
MLlib: Machine Learning in Apache Spark (2015, methodology): “The library also provides a number of low-level primitives and basic utilities for convex optimization, distributed linear algebra, statistical analysis, and feature extraction, and supports various I/O formats, including native support for LIBSVM format, data integration via Spark SQL (Armbrust et ”
Lakehouse: A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics (2021, methodology): “In addition, many ML systems have adopted DataFrames as the abstraction for manipulating data, and recent systems have designed declarative DataFrame APIs [11] that enable performing query optimizations for data accesses in ML workloads.”
Big data preprocessing: methods and prospects (2016, background): “This higher-level API is built on enhanced DataFrames structures [62].”
Opaque: An Oblivious and Encrypted Distributed Analytics Platform (2017, methodology): “While the techniques we present in this paper are general, we instantiate them using Apache Spark [4] by layering Opaque on top of Catalyst, the Spark SQL query optimizer (see Fig.”
Apache Spark (2016, background): “In Spark SQL, each record in an RDD holds a series of rows stored in binary format, and the system generates RDDs usually store only temporary data within an application, though some applications (such as the Spark SQL JDBC server) also share RDDs across multiple users.(2) Spark’s design as a storag”