BigBench: towards an industry standard benchmark for big data analytics

By A. Ghazal, T. Rabl, Minqing Hu, F. Raab, Meikel Poess, A. Crolotte, Hans-Arno Jacobsen. ACM SIGMOD Conference, 2013

437 citations recorded by the source corpus.

Subfields: Computer Networks and Communications, Information Systems

View Paper: https://www.semanticscholar.org/paper/f7bfcf58a57862b35022e7dd4954b8c15e731fb5

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

1. Kian-Lee Tan
2. Beng Chin Ooi
3. Ji Wang
4. Wei Wang
5. Rui Liu
6. Yanqiao Zhu
7. Tien Tuan Anh Dinh
8. Gang Chen

In Information Systems

1. Kian-Lee Tan
2. Beng Chin Ooi
3. Jianfeng Zhan
4. Ji Wang
5. Qiang Yang
6. Rui Liu
7. Yanqiao Zhu
8. Pan Lu

Citations in the corpus

• SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models (2023, positive): “BIG-Bench (Ghazal et al., 2013) offers a large-scale general-purpose test suite that requires 204 multiple-choice or exact-match tasks, and its extension BIG-Bench Hard (Suzgun et al., 2022) poses challenging CoT prompts.”
• BigBench V2: The New and Improved BigBench (2017, background): “Comparing and contrasting these benchmarks to BigBench, we identify the uniqueness and superiority of BigBench [5] as follows:”
• Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs (2023, methodology): “We evaluate the quality of confidence estimates across five types of reasoning tasks: 1) Commonsense Reasoning on two benchmarks, Sports Understanding (SportUND) (Kim, 2021) and StrategyQA (Geva et al., 2021) from BigBench (Ghazal et al., 2013); 2) Arithmetic Reasoning on two math problems, GSM8K (C”
• BigDataBench: A big data benchmark suite from internet services (2014, methodology): “This section presents our methodology and decisions on BigDataBench.”
• Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models (2023, methodology): “To evaluate general reasoning, we utilize Big-Bench-Hard (BBH (Suzgun et al., 2022)), incorporating 23 challenging tasks from Big-Bench (Ghazal et al., 2013).”
• BLOCKBENCH: A Framework for Analyzing Private Blockchains (2017, background): “HiBench [32] and BigBench [28] feature big-data analytics workloads for MapReduce-like systems.”