From LSAT: The Progress and Challenges of Complex Reasoning

By Siyuan Wang, Zhongkun Liu, Wanjun Zhong, Ming Zhou, Zhongyu Wei, Zhumin Chen, Nan Duan. 2021

64 citations recorded by the source corpus.

Complex reasoning aims to draw a correct inference based on complex rules. As a hallmark of human intelligence, it involves a degree of explicit reading comprehension, interpretation of logical knowledge and complex rule application. In this paper, we take a step forward in complex reasoning by systematically studying the three challenging and domain-general tasks of the Law School Admission Test (LSAT), including analytical reasoning, logical reasoning and reading comprehension. We propose a hybrid reasoning system to integrate these three tasks and achieve impressive overall performance on the LSAT tests. The experimental results demonstrate that our system endows itself a certain complex reasoning ability, especially the fundamental reading comprehension and challenging logical reasoning capacities. Further analysis also shows the effectiveness of combining the pre-trained models with the task-specific reasoning module, and integrating symbolic knowledge into discrete interpretable reasoning steps in complex reasoning. We further shed a light on the potential future directions, like unsupervised symbolic knowledge extraction, model interpretability, few-shot learning and comprehensive benchmark for complex reasoning.

Subfields: Artificial Intelligence, Political Science and International Relations, Prompting & Reasoning, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Fei Huang
2. Shumin Deng
3. Chuanqi Tan
4. Hanmeng Liu
5. Yuexin Zhang
6. Jian Liu
7. Shuofei Qiao
8. Yixin Ou

In Prompting & Reasoning

1. Fei Huang
2. Shumin Deng
3. Chuanqi Tan
4. Ningyu Zhang
5. Huajun Chen

In Evaluation & Benchmarks

1. Fei Huang
2. Chuanqi Tan
3. Hanmeng Liu
4. Jian Liu

Citations in the corpus

• Reasoning with Language Model Prompting: A Survey (2022, background): “what they are told or have already known (Duan et al., 2020; Wang et al., 2021; Bhargava and Ng, 2022).”
• Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4 (2023, background): “Figure 5 shows an example from the AR-LSAT test set.”