Supporting Scalable Analytics with Latency Constraints

By Boduo Li, Yanlei Diao, Prashant J. Shenoy. 2015

43 citations recorded by the source corpus.

Recently there has been a significant interest in building big data analytics systems that can handle both "big data" and "fast data". Our work is strongly motivated by recent real-world use cases that point to the need for a general, unified data processing framework to support analytical queries with different latency requirements. Toward this goal, we start with an analysis of existing big data systems to understand the causes of high latency. We then propose an extended architecture with mini-batches as granularity for computation and shuffling, and augment it with new model-driven resource allocation and runtime scheduling techniques to meet user latency requirements while maximizing throughput. Results from real-world workloads show that our techniques, implemented in Incremental Hadoop, reduce its latency from tens of seconds to sub-second, with 2x-5x increase in throughput. Our system also outperforms state-of-the-art distributed stream systems, Storm and Spark Streaming, by 1-2 orders of magnitude when combining latency and throughput.

Subfields: Computer Networks and Communications, Computer Vision and Pattern Recognition, Information Systems

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Computer Vision and Pattern Recognition

In Information Systems

Citations in the corpus

Deep Learning for IoT Big Data and Streaming Analytics: A Survey (2017, methodology): “The streaming data analytics on such frameworks is based on data parallelism and incremental processing [17].”