Analytical Reasoning of Text

By Wanjun Zhong, Siyuan Wang, Duyu Tang, Zenan Xu, Daya Guo, Yining Chen, Jiahai Wang, Jian Yin, Ming Zhou, Nan Duan. 2022

27 citations recorded by the source corpus.

Analytical reasoning is an essential and chal- 001 lenging task that requires a system to analyze 002 a scenario involving a set of particular circum- 003 stances and perform reasoning over it to make 004 conclusions. However, current neural models 005 with implicit reasoning ability struggle to solve 006 this task. In this paper, we study the challenge 007 of analytical reasoning of text and collect a new 008 dataset consisting of questions from the Law 009 School Admission Test from 1991 to 2016. We 010 analyze what knowledge understanding and rea- 011 soning abilities are required to do well on this 012 task, and present an approach dubbed ARM. 013 It extracts knowledge such as participants and 014 facts from the context. Such knowledge are 015 applied to an inference engine to deduce legit- 016 imate solutions for drawing conclusions. In 017 our experiments, we find that ubiquitous pre- 018 trained models struggle to deal with this task as 019 their performance is close to random guess. Re- 020 sults show that ARM outperforms pre-trained 021 models significantly. Moreover, we demon- 022 strate that ARM has better explicit interpretable 023 reasoning ability. 1 024

Subfields: Artificial Intelligence, Political Science and International Relations, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Nan Duan
2. Weizhu Chen
3. Shuai Lu
4. Yiduo Guo
5. Yaobo Liang
6. Wanjun Zhong
7. Jianfeng Gao
8. Mohit Bansal

In Natural Language Processing

1. Nan Duan
2. Weizhu Chen
3. Shuai Lu
4. Yaobo Liang
5. Wanjun Zhong
6. Jianfeng Gao
7. Xiaodong Liu
8. Mohit Bansal

Citations in the corpus

• AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models (2023, positive): “For the LSAT, we utilize data from Wang et al. (2022) and Zhong et al. (2022), which encompasses three tasks (logical reasoning, reading comprehension, and analytical reasoning) from the LSAT administered between 1991 and 2016.”