SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models

By Xiaoxuan Wang, Ziniu Hu, Pan Lu, Yanqiao Zhu, Jieyu Zhang, Satyen Subramaniam, Arjun R. Loomba, Shichang Zhang, Yizhou Sun, Wei Wang. 2023

252 citations recorded by the source corpus.

Most of the existing Large Language Model (LLM) benchmarks on scientific problem reasoning focus on problems grounded in high-school subjects and are confined to elementary algebraic operations. To systematically examine the reasoning capabilities required for solving complex scientific problems, we introduce an expansive benchmark suite SciBench for LLMs. SciBench contains a carefully curated dataset featuring a range of collegiate-level scientific problems from mathematics, chemistry, and physics domains. Based on the dataset, we conduct an in-depth benchmarking study of representative open-source and proprietary LLMs with various prompting strategies. The results reveal that the current LLMs fall short of delivering satisfactory performance, with the best overall score of merely 43.22%. Furthermore, through a detailed user study, we categorize the errors made by LLMs into ten problem-solving abilities. Our analysis indicates that no single prompting strategy significantly outperforms the others and some strategies that demonstrate improvements in certain problem-solving skills could result in declines in other skills. We envision that SciBench will catalyze further developments in the reasoning abilities of LLMs, thereby ultimately contributing to scientific research and discovery.

Subfields: Computer Science Applications, Developmental and Educational Psychology, Education, Large Language Models, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Science Applications

1. Pan Lu
2. Hannaneh Hajishirzi

In Developmental and Educational Psychology

1. Song-Chun Zhu
2. Jianfeng Gao
3. Hannaneh Hajishirzi
4. Kai-Wei Chang

In Education

1. Jianfeng Gao

In Large Language Models

1. Michel Galley
2. Yu Su
3. Jianfeng Gao
4. Hao Cheng
5. Kai-Wei Chang
6. Huan Sun
7. Song-Chun Zhu
8. Baolin Peng

In Evaluation & Benchmarks

1. Hannaneh Hajishirzi
2. Ying Nian Wu
3. Huan Sun
4. Song-Chun Zhu
5. Baolin Peng
6. Jianfeng Gao
7. Michel Galley
8. Ziyan Jiang

Citations in the corpus

• MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark (2024, methodology): “We then expand our dataset by incorporating questions from the STEM Website 2 , TheoremQA [10], and SciBench [40].”
• CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generation (2023, background): “For exam-4 ple, SciBench (Wang et al., 2023a), an evaluation benchmark for complex scientific problems, and C-Eval (Huang et al., 2023), an evaluation benchmark for multilevel and multi-discipline Chinese, strive to gather data that is difficult to extract or convert into text to mitigate this…”
• MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning (2023, methodology): “More recently, (Chen et al., 2023; Wang et al., 2023e) have proposed to tackle more challenging college-level science and math problems.”
• MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts (2023, background): “…text, such as QA, math, medicine, coding and science (Bubeck et al., 2023; Nori et al., 2023; Chen et al., 2021; Fu et al., 2023; Sun et al., 2023; Wang et al., 2023; Huang et al., 2023, 2022; Liu et al., 2023b On the vision-language side, there are several generative foundation models such as…”
• Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models (2023, background): “To address these shortcomings, there has been a growing interest in exploring the use of external tools and modular approaches to augment LLMs.”