Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4

By Hanmeng Liu, Ruoxi Ning, Zhiyang Teng, Jian Liu, Qiji Zhou, Yuexin Zhang. 2023

336 citations recorded by the source corpus.

Harnessing logical reasoning ability is a comprehensive natural language understanding endeavor. With the release of Generative Pretrained Transformer 4 (GPT-4), highlighted as"advanced"at reasoning tasks, we are eager to learn the GPT-4 performance on various logical reasoning tasks. This report analyses multiple logical reasoning datasets, with popular benchmarks like LogiQA and ReClor, and newly-released datasets like AR-LSAT. We test the multi-choice reading comprehension and natural language inference tasks with benchmarks requiring logical reasoning. We further construct a logical reasoning out-of-distribution dataset to investigate the robustness of ChatGPT and GPT-4. We also make a performance comparison between ChatGPT and GPT-4. Experiment results show that ChatGPT performs significantly better than the RoBERTa fine-tuning method on most logical reasoning benchmarks. With early access to the GPT-4 API we are able to conduct intense experiments on the GPT-4 model. The results show GPT-4 yields even higher performance on most logical reasoning datasets. Among benchmarks, ChatGPT and GPT-4 do relatively well on well-known datasets like LogiQA and ReClor. However, the performance drops significantly when handling newly released and out-of-distribution datasets. Logical reasoning remains challenging for ChatGPT and GPT-4, especially on out-of-distribution and natural language inference datasets. We release the prompt-style logical reasoning datasets as a benchmark suite and name it LogiEval.

Subfields: Artificial Intelligence, Large Language Models, Prompting & Reasoning, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Prompting & Reasoning

In Evaluation & Benchmarks

Citations in the corpus

Causal Reasoning and Large Language Models: Opening a New Frontier for Causality (2023, background): “This apparent capacity for both implicit and explicit consideration of causal factors has generated excitement towards understanding their reasoning capabilities (Hobbhahn et al., 2022; Kosoy et al., 2022; Willig et al., 2022; Liu et al., 2023; Zhang et al., 2023).”
PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts (2023, neutral): “Large language models (LLMs) have gained increasing popularity due to their unprecedented performance in various tasks such as sentiment analysis (Wang et al., 2019), question answering (Wang et al., 2019), logical reasoning (Liu et al., 2023a), etc.”
Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning (2023, background): “…with access to external tools and resources, such as the information retriever (Nakano et al., 2021; Shi et al., 2023; Lazaridou et al., 2022), calculator (Cobbe et al., 2021), code interpreter (Wang et al., 2022), planner (Liu et al., 2023a), and other pre-trained models (Shen et al., 2023).”
ChatGPT: The End of Online Exam Integrity? (2022, neutral): “It is also true that LLMs still encounter persistent difficulties across complex reasoning tasks [30] from which it is argued and claimed that due to this, they are again unsuitable for answering sophisticated exam questions with reliability [31,32].”
SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models (2023, background): “One percent of the speed of light is The momentum of the electron is given by The de Broglie wavelength of this electron is This wavelength is of atomic dimensions.”
How is ChatGPT's behavior changing over time? (2023, background): “…code has also been open-sourced. u Those papers focus on ML services for simple classification tasks such as sentiment analysis, while this work studies generative LLM services. s Chen et al., 2022), and studied how to efficiently estimate ML service performance shifts (L. (MLaaS) more broadly.”
A Survey on Evaluation of Large Language Models (2023, methodology): “[115] indicated that ChatGPT and GPT-4 outperform traditional fine-tuning methods on most benchmarks, demonstrating their superiority in logical reasoning.”