TheoremQA: A Theorem-driven Question Answering dataset

By Wenhu Chen, Ming Yin, Max W.F. Ku, Yixin Wan, Xueguang Ma, Jianyu Xu, Tony Xia, Xinyi Wang, Pan Lu. 2023

250 citations recorded by the source corpus.

The recent LLMs like GPT-4 and PaLM-2 have made tremendous progress in solving fundamental math problems like GSM8K by achieving over 90% accuracy. However, their capabilities to solve more challenging math problems which require domain-specific knowledge (i.e. theorem) have yet to be investigated. In this paper, we introduce TheoremQA, the first theorem-driven question-answering dataset designed to evaluate AI models' capabilities to apply theorems to solve challenging science problems. TheoremQA is curated by domain experts containing 800 high-quality questions covering 350 theorems (e.g. Taylor's theorem, Lagrange's theorem, Huffman coding, Quantum Theorem, Elasticity Theorem, etc) from Math, Physics, EE&CS, and Finance. We evaluate a wide spectrum of 16 large language and code models with different prompting strategies like Chain-of-Thoughts and Program-of-Thoughts. We found that GPT-4's capabilities to solve these problems are unparalleled, achieving an accuracy of 51% with Program-of-Thoughts Prompting. All the existing open-sourced models are below 15%, barely surpassing the random-guess baseline. Given the diversity and broad coverage of TheoremQA, we believe it can be used as a better benchmark to evaluate LLMs' capabilities to solve challenging science problems. The data and code are released in https://github.com/wenhuchen/TheoremQA.

Subfields: Artificial Intelligence, Information Systems, Question Answering & Reading Comprehension, Automated Theorem Proving & Formal Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Jianfeng Gao
  2. Zhiyuan Liu
  3. Mohit Bansal
  4. Maosong Sun
  5. Song-Chun Zhu
  6. Percy Liang
  7. Luke S. Zettlemoyer
  8. Yejin Choi

In Information Systems

  1. Maosong Sun
  2. Zhiyuan Liu
  3. Xu Han
  4. Xueguang Ma
  5. Huan Sun
  6. Fei Huang
  7. Rui Zhang
  8. Song-Chun Zhu

In Question Answering & Reading Comprehension

  1. Huan Sun
  2. Song-Chun Zhu
  3. Jingren Zhou
  4. Jianfeng Gao
  5. Pan Lu
  6. Wenhu Chen

In Automated Theorem Proving & Formal Methods

  1. Sean Welleck

Citations in the corpus