Discretized streams: fault-tolerant streaming computation at scale

By Matei A. Zaharia, Tathagata Das, Haoyuan Li, Timothy Hunter, Scott Shenker, Ion Stoica. Symposium on Operating Systems Principles, 2013

1,176 citations recorded by the source corpus.

Many "big data" applications must act on data in real time. Running these applications at ever-larger scales requires parallel platforms that automatically handle faults and stragglers. Unfortunately, current distributed stream processing models provide fault recovery in an expensive manner, requiring hot replication or long recovery times, and do not handle stragglers. We propose a new processing model, discretized streams (D-Streams), that overcomes these challenges. D-Streams enable a parallel recovery mechanism that improves efficiency over traditional replication and backup schemes, and tolerates stragglers. We show that they support a rich set of operators while attaining high per-node throughput similar to single-node systems, linear scaling to 100 nodes, sub-second latency, and sub-second fault recovery. Finally, D-Streams can easily be composed with batch and interactive query models like MapReduce, enabling rich applications that combine these modes. We implement D-Streams in a system called Spark Streaming.

Subfields: Computer Networks and Communications, Information Systems

View Paper: https://www.semanticscholar.org/paper/edac06ef9b2bc9062b7c92d740075e857c47eb88

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Information Systems

Citations in the corpus

MLlib: Machine Learning in Apache Spark (2015, methodology): “Additionally, Spark Streaming (Zaharia et al., 2013) allows users to process live data streams and thus enables the development of online learning algorithms, as in Freeman (2015).”
OneFlow: Redesign the Distributed Deep Learning Framework from Scratch (2021, background): “, 2020) and fine-grained fault resilience (Wang et al., 2021; Zaharia et al., 2013) besides the naive global checkpointing; (2) to implement auto placement and auto parallelism by designing a more efficient cost model, thus making it easier to use.”
Sonata: query-driven streaming network telemetry (2018, methodology): “These systems can benefit from horizontally scalable stream processors such as Spark Streaming [59] and Flink [38], but they also face scaling limitations due to packet parsing and cluster coordination [41].”
Resource Management in Fog/Edge Computing (2018, methodology): “uted on the node. For example, the underlying techniques of Apache Flink rely on batch processing of incoming data1. Similarly, Apache Spark2 employs the concept of Discretized Streams (or D-Streams) [21], a micro-batch processing technique that periodically performs batch computations over short ti”
Language-Directed Hardware Design for Network Performance Monitoring (2017, background): “Modern scale-out data-processing systems support 100K–1M operations per second per core [2, 4, 11, 43, 64], but processing every single packet (assuming a relatively large packet size of 1 KB) from a single 1 Tbit/s switch would need 100M operations per second — 2–3 orders of magnitude more than wha”
SparkNet: Training Deep Networks in Spark (2015, background): “…SparkNet, training a deep network on the output of a SQL query, or a graph computation, or a streaming data source is straightforward due to its general purpose nature and its support for SQL, graph computations, and data streams (Armbrust et al., 2015; Gonzalez et al., 2014; Zaharia et al., 2013).”
A Survey on Ensemble Learning for Data Stream Classification (2017, methodology): “16 StreamDM is an open-source framework for big data stream mining that uses the Spark Streaming [Zaharia et al. 2013] extension of the core Spark API (Application Program Interface).”
Naiad: a timely dataflow system (2013, methodology): “Context • Vertex centric computa/on models - Pregel [2] • TensorFlow [4] – uses /mely dataflow in dynamic computa/on • Straggler mi/ga/on a higher priority in some systems – RDD [5], D-Streams [6] (based on RDD).”
Stale View Cleaning: Getting Fresh Answers from Stale Materialized Views (2015, background): “Spark does have a streaming variant [51], however, this does not support the complex SQL derived materialized views used in this paper, and still relies on mini-batch updates.”
Live Video Analytics at Scale with Approximation and Delay-Tolerance (2017, background): “Production systems for stream processing like Storm [4], Stream- Scope [62], Flink [2], Trill [36] and Spark Streaming [89] allocate resources among multiple queries only based on resource fairness [7] [10] [27] [43] [51] common to cluster managers like Yarn [3] and Mesos [49].”
The Dataflow Model: A Practical Approach to Balancing Correctness, Latency, and Cost in Massive-Scale, Unbounded, Out-of-Order Data Processing (2015, background): “From the scale enabled by MapReduce [16] and its successors (e.g Hadoop [4], Pig [18], Hive [29], Spark [33]), to the vast body of work on streaming within the SQL community (e.g. query systems [1, 14, 15], windowing [22], data streams [24], time domains [28], semantic models [9]), to the more recen”
Tachyon: Reliable, Memory Speed Storage for Cluster Computing Frameworks (2014, methodology): “Previous frameworks that support long running workloads, such as Spark Streaming [54] and BAD-FS [16], circumvent this challenge by using periodic checkpointing.”