Twitter Heron: Stream Processing at Scale

By Sanjeev Kulkarni, Nikunj Bhagat, Maosong Fu, Vikas Kedigehalli, Christopher Kellogg, Sailesh Mittal, Jignesh M. Patel, K. Ramasamy, Siddarth Taneja. SIGMOD Conference, 2015

630 citations recorded by the source corpus.

Storm has long served as the main platform for real-time analytics at Twitter. However, as the scale of data being processed in real-time at Twitter has increased, along with an increase in the diversity and the number of use cases, many limitations of Storm have become apparent. We need a system that scales better, has better debug-ability, has better performance, and is easier to manage -- all while working in a shared cluster infrastructure. We considered various alternatives to meet these needs, and in the end concluded that we needed to build a new real-time stream data processing system. This paper presents the design and implementation of this new system, called Heron. Heron is now the de facto stream data processing engine inside Twitter, and in this paper we also share our experiences from running Heron in production. In this paper, we also provide empirical evidence demonstrating the efficiency and scalability of Heron.

Subfields: Computer Networks and Communications, Information Systems

View Paper: https://www.semanticscholar.org/paper/e847c3ec130da57328db79a7fea794b07dbccdd9

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Information Systems

Citations in the corpus

Benchmarking Streaming Computation Engines: Storm, Flink and Spark Streaming (2016, methodology): “For example, at Yahoo, the platform of choice has been Apache Storm, which replaced the internally developed S4 platform [8].”
Distributed data stream processing and edge computing: A survey on resource elasticity and future directions (2017, positive): “While maintaining API compatibility with Apache Storm, Twitter’s Heron [35] was built with a range of architectural improvements and mechanisms to achieve better e ﬃ ciency and to address several of Storm issues highlighted in previous work [56].”
MacroBase: Prioritizing Attention in Fast Data (2016, methodology): “me-series explanation operator [56], all without modifying the remainder of the operator pipeline. As this example illustrates, compared to conventional dataﬂoworiented analysis engines such as Heron [54] and Dryad [51], MacroBase provides users with a set of interfaces that are specialized for anal”
MacroBase: Analytic Monitoring for the Internet of Things (2016, methodology): “me-series explanation operator [56], all without modifying the remainder of the operator pipeline. As this example illustrates, compared to conventional dataﬂoworiented analysis engines such as Heron [54] and Dryad [51], MacroBase provides users with a set of interfaces that are specialized for anal”