BigDataBench: A big data benchmark suite from internet services

By Lei Wang, Jianfeng Zhan, Chunjie Luo, Yuqing Zhu, Qiang Yang, Yongqiang He, Wanling Gao, Zhen Jia, Yingjie Shi, Shujie Zhang and 5 more. International Symposium on High-Performance Computer Architecture, 2014

607 citations recorded by the source corpus.

As architecture, systems, and data management communities pay greater attention to innovative big data systems and architecture, the pressure of benchmarking and evaluating these systems rises. However, the complexity, diversity, frequently changed workloads, and rapid evolution of big data systems raise great challenges in big data benchmarking. Considering the broad use of big data systems, for the sake of fairness, big data benchmarks must include diversity of data and workloads, which is the prerequisite for evaluating big data systems and architecture. Most of the state-of-the-art big data benchmarking efforts target evaluating specific types of applications or system software stacks, and hence they are not qualified for serving the purposes mentioned above. This paper presents our joint research efforts on this issue with several industrial partners. Our big data benchmark suite-BigDataBench not only covers broad application scenarios, but also includes diverse and representative data sets. Currently, we choose 19 big data benchmarks from dimensions of application scenarios, operations/ algorithms, data types, data sources, software stacks, and application types, and they are comprehensive for fairly measuring and evaluating big data systems and architecture. BigDataBench is publicly available from the project home page http://prof.ict.ac.cn/BigDataBench. Also, we comprehensively characterize 19 big data workloads included in BigDataBench with varying data inputs. On a typical state-of-practice processor, Intel Xeon E5645, we have the following observations: First, in comparison with the traditional benchmarks: including PARSEC, HPCC, and SPECCPU, big data applications have very low operation intensity, which measures the ratio of the total number of instructions divided by the total byte number of memory accesses; Second, the volume of data input has non-negligible impact on micro-architecture characteristics, which may impose challenges for simulation-based big data architecture research; Last but not least, corroborating the observations in CloudSuite and DCBench (which use smaller data inputs), we find that the numbers of L1 instruction cache (L1I) misses per 1000 instructions (in short, MPKI) of the big data applications are higher than in the traditional benchmarks; also, we find that L3 caches are effective for the big data applications, corroborating the observation in DCBench.

Subfields: Computer Networks and Communications, Hardware and Architecture, Information Systems

View Paper: https://www.semanticscholar.org/paper/e3642ab645a3fdcce97f854503ba67f4b503b9ea

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Hardware and Architecture

In Information Systems

Citations in the corpus

Processing Data Where It Makes Sense: Enabling In-Memory Computation (2019, background): “Across all of these systems, including servers, cloud platforms, and mobile/embedded devices, the data working set sizes of modern applications are rapidly growing, causing the main memory to be a significant bottleneck for these applications [1, 2, 3, 4, 5, 6, 7].”
Characterizing, Modeling, and Benchmarking RocksDB Key-Value Workloads at Facebook (2020, neutral): “Several benchmarks designed for big data NoSQL systems, such as YCSB [11], LinkBench [4], and BigDataBench [45], are also widely used to evaluate KV-store performance.”
A new direction to promote the implementation of artificial intelligence in natural clinical settings (2019, background): “Wang et al. 41 presented a comprehensive big data benchmark suite, named Big-DataBench, for the architecture and system evaluation.”
BigBench V2: The New and Improved BigBench (2017, background): “More generic benchmarks include BigFrame [24], PRIMEBALL [25], and BigDataBench [26].”
An Open-Source Benchmark Suite for Microservices and Their Hardware-Software Implications for Cloud & Edge Systems (2019, background): “Cloud applications have attracted a lot of attention over the past decade, with several benchmark suites being released both from academia and industry [37, 44, 51, 81, 88].”